[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260505 20:17:54.363893 28669 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.277s user 0.495s sys 0.780s
W20260505 20:17:54.364434 28669 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.278s user 0.495s sys 0.780s
I20260505 20:17:54.379887 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:41931
I20260505 20:17:54.382079 28669 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260505 20:17:54.384593 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:17:54.406098 28680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:17:54.409947 28681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:17:54.411510 28683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:17:54.530411 28669 server_base.cc:1061] running on GCE node
I20260505 20:17:54.531780 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:17:54.532014 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:17:54.532315 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012274532284 us; error 0 us; skew 500 ppm
I20260505 20:17:54.533136 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:17:54.540606 28669 webserver.cc:492] Webserver started at http://127.27.255.126:45991/ using document root <none> and password file <none>
I20260505 20:17:54.541693 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:17:54.541941 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:17:54.542467 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:17:54.548708 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "dad512586c8346ebbedf2e84c3bb260c"
format_stamp: "Formatted at 2026-05-05 20:17:54 on dist-test-slave-dnnj"
I20260505 20:17:54.559922 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260505 20:17:54.567111 28688 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:17:54.568933 28669 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260505 20:17:54.569569 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "dad512586c8346ebbedf2e84c3bb260c"
format_stamp: "Formatted at 2026-05-05 20:17:54 on dist-test-slave-dnnj"
I20260505 20:17:54.569926 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:17:54.593050 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:17:54.594748 28669 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260505 20:17:54.595265 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:17:54.684525 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:41931
I20260505 20:17:54.684640 28739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:41931 every 8 connection(s)
I20260505 20:17:54.691864 28740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:17:54.719841 28740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c: Bootstrap starting.
I20260505 20:17:54.728891 28740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c: Neither blocks nor log segments found. Creating new log.
I20260505 20:17:54.731277 28740 log.cc:826] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c: Log is configured to *not* fsync() on all Append() calls
I20260505 20:17:54.737371 28740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c: No bootstrap required, opened a new log
I20260505 20:17:54.761343 28740 raft_consensus.cc:359] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad512586c8346ebbedf2e84c3bb260c" member_type: VOTER }
I20260505 20:17:54.762032 28740 raft_consensus.cc:385] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:17:54.762331 28740 raft_consensus.cc:740] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad512586c8346ebbedf2e84c3bb260c, State: Initialized, Role: FOLLOWER
I20260505 20:17:54.763159 28740 consensus_queue.cc:260] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad512586c8346ebbedf2e84c3bb260c" member_type: VOTER }
I20260505 20:17:54.763741 28740 raft_consensus.cc:399] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:17:54.764045 28740 raft_consensus.cc:493] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:17:54.764415 28740 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:17:54.770682 28740 raft_consensus.cc:515] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad512586c8346ebbedf2e84c3bb260c" member_type: VOTER }
I20260505 20:17:54.772004 28740 leader_election.cc:304] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dad512586c8346ebbedf2e84c3bb260c; no voters:
I20260505 20:17:54.774578 28740 leader_election.cc:290] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:17:54.774999 28743 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:17:54.777472 28743 raft_consensus.cc:697] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 1 LEADER]: Becoming Leader. State: Replica: dad512586c8346ebbedf2e84c3bb260c, State: Running, Role: LEADER
I20260505 20:17:54.778626 28743 consensus_queue.cc:237] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad512586c8346ebbedf2e84c3bb260c" member_type: VOTER }
I20260505 20:17:54.779560 28740 sys_catalog.cc:565] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:17:54.789682 28745 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dad512586c8346ebbedf2e84c3bb260c. Latest consensus state: current_term: 1 leader_uuid: "dad512586c8346ebbedf2e84c3bb260c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad512586c8346ebbedf2e84c3bb260c" member_type: VOTER } }
I20260505 20:17:54.790876 28745 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [sys.catalog]: This master's current role is: LEADER
I20260505 20:17:54.790602 28744 sys_catalog.cc:455] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dad512586c8346ebbedf2e84c3bb260c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dad512586c8346ebbedf2e84c3bb260c" member_type: VOTER } }
I20260505 20:17:54.792392 28744 sys_catalog.cc:458] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [sys.catalog]: This master's current role is: LEADER
I20260505 20:17:54.800513 28753 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:17:54.813639 28753 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:17:54.817122 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:17:54.834549 28753 catalog_manager.cc:1357] Generated new cluster ID: 1e67fcf58fc241b6b21bec674434d88d
I20260505 20:17:54.834887 28753 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:17:54.860783 28753 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:17:54.863831 28753 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:17:54.894135 28753 catalog_manager.cc:6044] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c: Generated new TSK 0
I20260505 20:17:54.895639 28753 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:17:54.950035 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:17:54.961452 28762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:17:54.962939 28763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:17:54.969241 28765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:17:54.970456 28669 server_base.cc:1061] running on GCE node
I20260505 20:17:54.972198 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:17:54.972496 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:17:54.972733 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012274972712 us; error 0 us; skew 500 ppm
I20260505 20:17:54.973830 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:17:54.980105 28669 webserver.cc:492] Webserver started at http://127.27.255.65:40533/ using document root <none> and password file <none>
I20260505 20:17:54.980916 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:17:54.981170 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:17:54.981635 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:17:54.983012 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "d64549c796c04447a9f28e704ff839d5"
format_stamp: "Formatted at 2026-05-05 20:17:54 on dist-test-slave-dnnj"
I20260505 20:17:54.989071 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260505 20:17:54.994445 28770 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:17:54.997090 28669 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20260505 20:17:54.997686 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "d64549c796c04447a9f28e704ff839d5"
format_stamp: "Formatted at 2026-05-05 20:17:54 on dist-test-slave-dnnj"
I20260505 20:17:54.998023 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:17:55.059374 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:17:55.061712 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:17:55.065001 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:17:55.069155 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:17:55.069411 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:17:55.069823 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:17:55.070004 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:17:55.188019 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:38047
I20260505 20:17:55.188095 28832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:38047 every 8 connection(s)
I20260505 20:17:55.196784 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:17:55.208031 28837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:17:55.211309 28838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:17:55.221807 28840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:17:55.223780 28669 server_base.cc:1061] running on GCE node
I20260505 20:17:55.224748 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:17:55.224983 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:17:55.225172 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012275225152 us; error 0 us; skew 500 ppm
I20260505 20:17:55.225905 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:17:55.231575 28669 webserver.cc:492] Webserver started at http://127.27.255.66:45907/ using document root <none> and password file <none>
I20260505 20:17:55.232267 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:17:55.232538 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:17:55.232858 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:17:55.232955 28833 heartbeater.cc:344] Connected to a master server at 127.27.255.126:41931
I20260505 20:17:55.234138 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/ts-1-root/instance:
uuid: "31ef237e0d434b599dce76b5a563c3af"
format_stamp: "Formatted at 2026-05-05 20:17:55 on dist-test-slave-dnnj"
I20260505 20:17:55.236831 28833 heartbeater.cc:461] Registering TS with master...
I20260505 20:17:55.240777 28833 heartbeater.cc:507] Master 127.27.255.126:41931 requested a full tablet report, sending...
I20260505 20:17:55.246589 28705 ts_manager.cc:194] Registered new tserver with Master: d64549c796c04447a9f28e704ff839d5 (127.27.255.65:38047)
I20260505 20:17:55.249783 28705 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39636
I20260505 20:17:55.250735 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260505 20:17:55.256359 28845 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:17:55.257325 28669 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260505 20:17:55.257876 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/ts-1-root
uuid: "31ef237e0d434b599dce76b5a563c3af"
format_stamp: "Formatted at 2026-05-05 20:17:55 on dist-test-slave-dnnj"
I20260505 20:17:55.258275 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1778012272869005-28669-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:17:55.288928 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:17:55.291143 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:17:55.294237 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:17:55.298758 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:17:55.299093 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:17:55.299520 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:17:55.299762 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:17:55.353722 28833 heartbeater.cc:499] Master 127.27.255.126:41931 was elected leader, sending a full tablet report...
I20260505 20:17:55.402202 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.66:37529
I20260505 20:17:55.402269 28907 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.66:37529 every 8 connection(s)
I20260505 20:17:55.421725 28908 heartbeater.cc:344] Connected to a master server at 127.27.255.126:41931
I20260505 20:17:55.422148 28908 heartbeater.cc:461] Registering TS with master...
I20260505 20:17:55.422936 28908 heartbeater.cc:507] Master 127.27.255.126:41931 requested a full tablet report, sending...
I20260505 20:17:55.425012 28705 ts_manager.cc:194] Registered new tserver with Master: 31ef237e0d434b599dce76b5a563c3af (127.27.255.66:37529)
I20260505 20:17:55.425740 28669 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.018017756s
I20260505 20:17:55.426780 28705 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39650
I20260505 20:17:55.472894 28705 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20260505 20:17:55.500358 28705 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260505 20:17:55.529809 28908 heartbeater.cc:499] Master 127.27.255.126:41931 was elected leader, sending a full tablet report...
I20260505 20:17:55.581073 28798 tablet_service.cc:1511] Processing CreateTablet for tablet 2dee1628690f4351a5376902199f35f2 (DEFAULT_TABLE table=test-table [id=19686bbf83d34daab12fc1ca194a7066]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260505 20:17:55.582682 28798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dee1628690f4351a5376902199f35f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:17:55.584424 28873 tablet_service.cc:1511] Processing CreateTablet for tablet 2dee1628690f4351a5376902199f35f2 (DEFAULT_TABLE table=test-table [id=19686bbf83d34daab12fc1ca194a7066]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260505 20:17:55.585984 28873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dee1628690f4351a5376902199f35f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:17:55.591096 28872 tablet_service.cc:1511] Processing CreateTablet for tablet 5f5ae977b6044246a068a6000f155c84 (DEFAULT_TABLE table=test-table [id=19686bbf83d34daab12fc1ca194a7066]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260505 20:17:55.591928 28797 tablet_service.cc:1511] Processing CreateTablet for tablet 5f5ae977b6044246a068a6000f155c84 (DEFAULT_TABLE table=test-table [id=19686bbf83d34daab12fc1ca194a7066]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260505 20:17:55.593252 28872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5ae977b6044246a068a6000f155c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:17:55.593668 28797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5ae977b6044246a068a6000f155c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:17:55.605485 28924 tablet_bootstrap.cc:492] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5: Bootstrap starting.
I20260505 20:17:55.623831 28925 tablet_bootstrap.cc:492] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af: Bootstrap starting.
I20260505 20:17:55.626730 28924 tablet_bootstrap.cc:654] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5: Neither blocks nor log segments found. Creating new log.
I20260505 20:17:55.635237 28925 tablet_bootstrap.cc:654] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af: Neither blocks nor log segments found. Creating new log.
I20260505 20:17:55.644532 28924 tablet_bootstrap.cc:492] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5: No bootstrap required, opened a new log
I20260505 20:17:55.645015 28925 tablet_bootstrap.cc:492] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af: No bootstrap required, opened a new log
I20260505 20:17:55.645099 28924 ts_tablet_manager.cc:1403] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5: Time spent bootstrapping tablet: real 0.040s user 0.022s sys 0.012s
I20260505 20:17:55.645783 28925 ts_tablet_manager.cc:1403] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.007s
I20260505 20:17:55.649374 28924 raft_consensus.cc:359] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:55.650049 28924 raft_consensus.cc:740] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d64549c796c04447a9f28e704ff839d5, State: Initialized, Role: FOLLOWER
I20260505 20:17:55.649654 28925 raft_consensus.cc:359] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:55.650909 28924 consensus_queue.cc:260] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:55.651556 28925 raft_consensus.cc:740] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ef237e0d434b599dce76b5a563c3af, State: Initialized, Role: FOLLOWER
I20260505 20:17:55.652374 28925 consensus_queue.cc:260] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:55.656680 28924 ts_tablet_manager.cc:1434] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260505 20:17:55.659900 28924 tablet_bootstrap.cc:492] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5: Bootstrap starting.
I20260505 20:17:55.664204 28925 ts_tablet_manager.cc:1434] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af: Time spent starting tablet: real 0.017s user 0.009s sys 0.005s
I20260505 20:17:55.670961 28924 tablet_bootstrap.cc:654] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5: Neither blocks nor log segments found. Creating new log.
I20260505 20:17:55.674559 28925 tablet_bootstrap.cc:492] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af: Bootstrap starting.
I20260505 20:17:55.684929 28925 tablet_bootstrap.cc:654] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af: Neither blocks nor log segments found. Creating new log.
W20260505 20:17:55.696480 28834 tablet.cc:2404] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260505 20:17:55.701344 28925 tablet_bootstrap.cc:492] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af: No bootstrap required, opened a new log
I20260505 20:17:55.702040 28925 ts_tablet_manager.cc:1403] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.012s
I20260505 20:17:55.707216 28924 tablet_bootstrap.cc:492] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5: No bootstrap required, opened a new log
I20260505 20:17:55.707846 28924 ts_tablet_manager.cc:1403] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5: Time spent bootstrapping tablet: real 0.050s user 0.022s sys 0.024s
I20260505 20:17:55.707949 28925 raft_consensus.cc:359] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } } peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } }
I20260505 20:17:55.709105 28925 raft_consensus.cc:740] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ef237e0d434b599dce76b5a563c3af, State: Initialized, Role: FOLLOWER
I20260505 20:17:55.709933 28925 consensus_queue.cc:260] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } } peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } }
I20260505 20:17:55.712155 28925 ts_tablet_manager.cc:1434] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260505 20:17:55.712318 28924 raft_consensus.cc:359] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } } peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } }
I20260505 20:17:55.713019 28924 raft_consensus.cc:740] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d64549c796c04447a9f28e704ff839d5, State: Initialized, Role: FOLLOWER
I20260505 20:17:55.713667 28924 consensus_queue.cc:260] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } } peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } }
I20260505 20:17:55.716964 28924 ts_tablet_manager.cc:1434] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260505 20:17:55.784698 28669 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20260505 20:17:55.785058 28669 raft_consensus.cc:423] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 0 FOLLOWER]: Emulating election...
I20260505 20:17:55.785342 28669 raft_consensus.cc:3060] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:17:55.791519 28669 raft_consensus.cc:697] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 1 LEADER]: Becoming Leader. State: Replica: 31ef237e0d434b599dce76b5a563c3af, State: Running, Role: LEADER
I20260505 20:17:55.792372 28669 consensus_queue.cc:237] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } } peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } }
I20260505 20:17:55.797838 28669 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260505 20:17:55.804795 28704 catalog_manager.cc:5671] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af reported cstate change: term changed from 0 to 1, leader changed from <none> to 31ef237e0d434b599dce76b5a563c3af (127.27.255.66). New cstate: current_term: 1 leader_uuid: "31ef237e0d434b599dce76b5a563c3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } health_report { overall_health: HEALTHY } } }
I20260505 20:17:55.826421 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260505 20:17:55.932684 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260505 20:17:56.138447 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260505 20:17:56.394382 28808 raft_consensus.cc:3060] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:17:56.404443 28935 consensus_queue.cc:1048] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260505 20:17:56.443389 28669 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260505 20:17:56.443672 28669 raft_consensus.cc:423] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 0 FOLLOWER]: Emulating election...
I20260505 20:17:56.443917 28669 raft_consensus.cc:3060] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:17:56.448573 28669 raft_consensus.cc:697] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 1 LEADER]: Becoming Leader. State: Replica: 31ef237e0d434b599dce76b5a563c3af, State: Running, Role: LEADER
I20260505 20:17:56.450091 28669 consensus_queue.cc:237] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:56.454530 28669 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260505 20:17:56.459697 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260505 20:17:56.460351 28704 catalog_manager.cc:5671] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af reported cstate change: term changed from 0 to 1, leader changed from <none> to 31ef237e0d434b599dce76b5a563c3af (127.27.255.66). New cstate: current_term: 1 leader_uuid: "31ef237e0d434b599dce76b5a563c3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } health_report { overall_health: UNKNOWN } } }
I20260505 20:17:56.565048 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260505 20:17:56.771081 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260505 20:17:56.888835 28808 raft_consensus.cc:3060] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:17:56.897156 28935 consensus_queue.cc:1048] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260505 20:17:57.276996 28669 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260505 20:17:57.277253 28669 raft_consensus.cc:423] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 1 LEADER]: Emulating election...
I20260505 20:17:57.277585 28669 raft_consensus.cc:3055] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 1 LEADER]: Stepping down as leader of term 1
I20260505 20:17:57.277776 28669 raft_consensus.cc:740] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31ef237e0d434b599dce76b5a563c3af, State: Running, Role: LEADER
I20260505 20:17:57.278306 28669 consensus_queue.cc:260] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:57.279168 28669 raft_consensus.cc:3060] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:17:57.283627 28669 raft_consensus.cc:697] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 2 LEADER]: Becoming Leader. State: Replica: 31ef237e0d434b599dce76b5a563c3af, State: Running, Role: LEADER
I20260505 20:17:57.284349 28669 consensus_queue.cc:237] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:57.286459 28669 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260505 20:17:57.289099 28704 catalog_manager.cc:5671] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "31ef237e0d434b599dce76b5a563c3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } health_report { overall_health: UNKNOWN } } }
I20260505 20:17:57.290288 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260505 20:17:57.395699 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260505 20:17:57.600754 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260505 20:17:57.698668 28808 raft_consensus.cc:3060] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:17:57.704540 28808 raft_consensus.cc:1275] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 2 FOLLOWER]: Refusing update from remote peer 31ef237e0d434b599dce76b5a563c3af: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260505 20:17:57.706183 28935 consensus_queue.cc:1048] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260505 20:17:57.906247 28669 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260505 20:17:57.906685 28669 raft_consensus.cc:423] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [term 1 FOLLOWER]: Emulating election...
I20260505 20:17:57.906996 28669 raft_consensus.cc:3060] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:17:57.912724 28669 raft_consensus.cc:697] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [term 2 LEADER]: Becoming Leader. State: Replica: d64549c796c04447a9f28e704ff839d5, State: Running, Role: LEADER
I20260505 20:17:57.913426 28669 consensus_queue.cc:237] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } } peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } }
I20260505 20:17:57.915767 28669 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260505 20:17:57.920626 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260505 20:17:57.920760 28705 catalog_manager.cc:5671] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 reported cstate change: term changed from 1 to 2, leader changed from 31ef237e0d434b599dce76b5a563c3af (127.27.255.66) to d64549c796c04447a9f28e704ff839d5 (127.27.255.65). New cstate: current_term: 2 leader_uuid: "d64549c796c04447a9f28e704ff839d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } health_report { overall_health: UNKNOWN } } }
I20260505 20:17:58.025414 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260505 20:17:58.130769 28808 raft_consensus.cc:1240] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [term 2 LEADER]: Rejecting Update request from peer 31ef237e0d434b599dce76b5a563c3af for earlier term 1. Current term is 2. Ops: []
I20260505 20:17:58.132285 28954 consensus_queue.cc:1059] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260505 20:17:58.133118 28954 consensus_queue.cc:1243] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Peer d64549c796c04447a9f28e704ff839d5 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260505 20:17:58.133519 28935 raft_consensus.cc:3055] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 1 LEADER]: Stepping down as leader of term 1
I20260505 20:17:58.133767 28935 raft_consensus.cc:740] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31ef237e0d434b599dce76b5a563c3af, State: Running, Role: LEADER
I20260505 20:17:58.134229 28935 consensus_queue.cc:260] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } } peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } }
I20260505 20:17:58.134919 28935 raft_consensus.cc:3060] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:17:58.230654 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260505 20:17:58.383741 28883 raft_consensus.cc:1275] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 2 FOLLOWER]: Refusing update from remote peer d64549c796c04447a9f28e704ff839d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260505 20:17:58.385169 28951 consensus_queue.cc:1048] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260505 20:17:58.736176 28669 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20260505 20:17:58.736663 28669 raft_consensus.cc:423] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 2 FOLLOWER]: Emulating election...
I20260505 20:17:58.736940 28669 raft_consensus.cc:3060] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:17:58.742105 28669 raft_consensus.cc:697] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 3 LEADER]: Becoming Leader. State: Replica: d64549c796c04447a9f28e704ff839d5, State: Running, Role: LEADER
I20260505 20:17:58.742859 28669 consensus_queue.cc:237] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:58.745225 28669 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260505 20:17:58.749878 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260505 20:17:58.749306 28703 catalog_manager.cc:5671] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 reported cstate change: term changed from 2 to 3, leader changed from 31ef237e0d434b599dce76b5a563c3af (127.27.255.66) to d64549c796c04447a9f28e704ff839d5 (127.27.255.65). New cstate: current_term: 3 leader_uuid: "d64549c796c04447a9f28e704ff839d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } health_report { overall_health: HEALTHY } } }
I20260505 20:17:58.854977 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260505 20:17:58.916870 28808 raft_consensus.cc:1240] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 3 LEADER]: Rejecting Update request from peer 31ef237e0d434b599dce76b5a563c3af for earlier term 2. Current term is 3. Ops: []
I20260505 20:17:58.918624 28970 consensus_queue.cc:1059] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260505 20:17:58.919893 28970 consensus_queue.cc:1243] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Peer d64549c796c04447a9f28e704ff839d5 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260505 20:17:58.920521 28971 raft_consensus.cc:3055] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 2 LEADER]: Stepping down as leader of term 2
I20260505 20:17:58.920879 28971 raft_consensus.cc:740] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 31ef237e0d434b599dce76b5a563c3af, State: Running, Role: LEADER
I20260505 20:17:58.921484 28971 consensus_queue.cc:260] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:58.922427 28971 raft_consensus.cc:3060] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:17:59.060387 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260505 20:17:59.168459 28883 raft_consensus.cc:1275] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 3 FOLLOWER]: Refusing update from remote peer d64549c796c04447a9f28e704ff839d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260505 20:17:59.169914 28951 consensus_queue.cc:1048] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260505 20:17:59.365916 28669 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260505 20:17:59.366284 28669 raft_consensus.cc:423] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 3 FOLLOWER]: Emulating election...
I20260505 20:17:59.366570 28669 raft_consensus.cc:3060] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 3 FOLLOWER]: Advancing to term 4
I20260505 20:17:59.373943 28669 raft_consensus.cc:697] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 4 LEADER]: Becoming Leader. State: Replica: 31ef237e0d434b599dce76b5a563c3af, State: Running, Role: LEADER
I20260505 20:17:59.374714 28669 consensus_queue.cc:237] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:59.376417 28669 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260505 20:17:59.381650 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260505 20:17:59.381659 28704 catalog_manager.cc:5671] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af reported cstate change: term changed from 3 to 4, leader changed from d64549c796c04447a9f28e704ff839d5 (127.27.255.65) to 31ef237e0d434b599dce76b5a563c3af (127.27.255.66). New cstate: current_term: 4 leader_uuid: "31ef237e0d434b599dce76b5a563c3af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } health_report { overall_health: UNKNOWN } } }
I20260505 20:17:59.486385 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260505 20:17:59.572094 28883 raft_consensus.cc:1240] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 4 LEADER]: Rejecting Update request from peer d64549c796c04447a9f28e704ff839d5 for earlier term 3. Current term is 4. Ops: []
I20260505 20:17:59.573757 28973 consensus_queue.cc:1059] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260505 20:17:59.574443 28973 consensus_queue.cc:1243] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [LEADER]: Peer 31ef237e0d434b599dce76b5a563c3af log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260505 20:17:59.574842 28951 raft_consensus.cc:3055] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 3 LEADER]: Stepping down as leader of term 3
I20260505 20:17:59.575109 28951 raft_consensus.cc:740] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d64549c796c04447a9f28e704ff839d5, State: Running, Role: LEADER
I20260505 20:17:59.575600 28951 consensus_queue.cc:260] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31ef237e0d434b599dce76b5a563c3af" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37529 } } peers { permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 } }
I20260505 20:17:59.576340 28951 raft_consensus.cc:3060] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 3 FOLLOWER]: Advancing to term 4
I20260505 20:17:59.690966 28669 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260505 20:17:59.925370 28808 raft_consensus.cc:1275] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 4 FOLLOWER]: Refusing update from remote peer 31ef237e0d434b599dce76b5a563c3af: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260505 20:17:59.927021 28979 consensus_queue.cc:1048] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d64549c796c04447a9f28e704ff839d5" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 38047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260505 20:18:00.198668 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:00.225411 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:00.226444 28669 tablet_replica.cc:333] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5: stopping tablet replica
I20260505 20:18:00.227180 28669 raft_consensus.cc:2243] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [term 2 LEADER]: Raft consensus shutting down.
I20260505 20:18:00.227921 28669 raft_consensus.cc:2272] T 5f5ae977b6044246a068a6000f155c84 P d64549c796c04447a9f28e704ff839d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:00.230412 28669 tablet_replica.cc:333] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5: stopping tablet replica
I20260505 20:18:00.230932 28669 raft_consensus.cc:2243] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:00.231488 28669 raft_consensus.cc:2272] T 2dee1628690f4351a5376902199f35f2 P d64549c796c04447a9f28e704ff839d5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:00.334555 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:00.345698 28669 tablet_server.cc:179] TabletServer@127.27.255.66:0 shutting down...
I20260505 20:18:00.375021 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:00.375876 28669 tablet_replica.cc:333] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af: stopping tablet replica
I20260505 20:18:00.376528 28669 raft_consensus.cc:2243] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 2 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:00.377019 28669 raft_consensus.cc:2272] T 5f5ae977b6044246a068a6000f155c84 P 31ef237e0d434b599dce76b5a563c3af [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:00.378836 28669 tablet_replica.cc:333] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af: stopping tablet replica
I20260505 20:18:00.379320 28669 raft_consensus.cc:2243] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 4 LEADER]: Raft consensus shutting down.
I20260505 20:18:00.379976 28669 raft_consensus.cc:2272] T 2dee1628690f4351a5376902199f35f2 P 31ef237e0d434b599dce76b5a563c3af [term 4 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:00.391357 28669 tablet_server.cc:196] TabletServer@127.27.255.66:0 shutdown complete.
I20260505 20:18:00.403534 28669 master.cc:562] Master@127.27.255.126:41931 shutting down...
I20260505 20:18:00.571838 28760 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20260505 20:18:00.582094 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:00.582857 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:00.583171 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P dad512586c8346ebbedf2e84c3bb260c: stopping tablet replica
I20260505 20:18:00.603116 28669 master.cc:584] Master@127.27.255.126:41931 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (7592 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20260505 20:18:00.637010 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:42717
I20260505 20:18:00.638257 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:00.644770 28987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:00.644879 28988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:00.646312 28990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:00.647964 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:00.649593 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:00.649834 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:00.650135 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012280650073 us; error 0 us; skew 500 ppm
I20260505 20:18:00.650853 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:00.653913 28669 webserver.cc:492] Webserver started at http://127.27.255.126:38049/ using document root <none> and password file <none>
I20260505 20:18:00.654472 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:00.654681 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:00.655007 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:00.656286 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "27e8ee6206614f1dbf02327792bf6ee2"
format_stamp: "Formatted at 2026-05-05 20:18:00 on dist-test-slave-dnnj"
I20260505 20:18:00.662533 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260505 20:18:00.667131 28995 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:00.668139 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260505 20:18:00.668717 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "27e8ee6206614f1dbf02327792bf6ee2"
format_stamp: "Formatted at 2026-05-05 20:18:00 on dist-test-slave-dnnj"
I20260505 20:18:00.669063 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:00.686344 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:00.687563 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:00.730721 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:42717
I20260505 20:18:00.730840 29046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:42717 every 8 connection(s)
I20260505 20:18:00.735572 29047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:00.747103 29047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2: Bootstrap starting.
I20260505 20:18:00.751787 29047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:00.756520 29047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2: No bootstrap required, opened a new log
I20260505 20:18:00.759202 29047 raft_consensus.cc:359] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e8ee6206614f1dbf02327792bf6ee2" member_type: VOTER }
I20260505 20:18:00.759702 29047 raft_consensus.cc:385] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:00.759895 29047 raft_consensus.cc:740] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e8ee6206614f1dbf02327792bf6ee2, State: Initialized, Role: FOLLOWER
I20260505 20:18:00.760391 29047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e8ee6206614f1dbf02327792bf6ee2" member_type: VOTER }
I20260505 20:18:00.760856 29047 raft_consensus.cc:399] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:00.761051 29047 raft_consensus.cc:493] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:00.761277 29047 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:00.766535 29047 raft_consensus.cc:515] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e8ee6206614f1dbf02327792bf6ee2" member_type: VOTER }
I20260505 20:18:00.767179 29047 leader_election.cc:304] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27e8ee6206614f1dbf02327792bf6ee2; no voters:
I20260505 20:18:00.768863 29047 leader_election.cc:290] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:00.769801 29050 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:00.771579 29050 raft_consensus.cc:697] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 1 LEADER]: Becoming Leader. State: Replica: 27e8ee6206614f1dbf02327792bf6ee2, State: Running, Role: LEADER
I20260505 20:18:00.772413 29050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e8ee6206614f1dbf02327792bf6ee2" member_type: VOTER }
I20260505 20:18:00.773626 29047 sys_catalog.cc:565] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:00.775918 29051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27e8ee6206614f1dbf02327792bf6ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e8ee6206614f1dbf02327792bf6ee2" member_type: VOTER } }
I20260505 20:18:00.775934 29052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27e8ee6206614f1dbf02327792bf6ee2. Latest consensus state: current_term: 1 leader_uuid: "27e8ee6206614f1dbf02327792bf6ee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27e8ee6206614f1dbf02327792bf6ee2" member_type: VOTER } }
I20260505 20:18:00.777058 29051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:00.777148 29052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:00.782624 29055 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:00.791734 29055 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:00.792906 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:00.803622 29055 catalog_manager.cc:1357] Generated new cluster ID: 31122601d7c24b648c2451bbf8d429a7
I20260505 20:18:00.803951 29055 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:00.832136 29055 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:00.833739 29055 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:00.852032 29055 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2: Generated new TSK 0
I20260505 20:18:00.852835 29055 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:00.860538 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:00.867388 29068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:00.868257 29069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:00.870733 29071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:00.871439 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:00.872371 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:00.872625 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:00.872807 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012280872783 us; error 0 us; skew 500 ppm
I20260505 20:18:00.873371 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:00.876192 28669 webserver.cc:492] Webserver started at http://127.27.255.65:41983/ using document root <none> and password file <none>
I20260505 20:18:00.876760 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:00.876989 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:00.877327 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:00.878682 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "19b5c4a3f26e40419ed87526f4873650"
format_stamp: "Formatted at 2026-05-05 20:18:00 on dist-test-slave-dnnj"
I20260505 20:18:00.883912 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260505 20:18:00.888016 29076 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:00.889086 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:00.889361 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "19b5c4a3f26e40419ed87526f4873650"
format_stamp: "Formatted at 2026-05-05 20:18:00 on dist-test-slave-dnnj"
I20260505 20:18:00.889729 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:00.903012 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:00.904408 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:00.906222 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:00.909039 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:00.909252 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:00.909682 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:00.909874 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:00.959379 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:33599
I20260505 20:18:00.959461 29138 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:33599 every 8 connection(s)
I20260505 20:18:00.964823 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:00.972079 29143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:00.973434 29144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:00.977947 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:00.977869 29146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:00.979377 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:00.979679 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:00.979893 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012280979871 us; error 0 us; skew 500 ppm
I20260505 20:18:00.980667 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:00.982265 29139 heartbeater.cc:344] Connected to a master server at 127.27.255.126:42717
I20260505 20:18:00.982653 29139 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:00.983471 29139 heartbeater.cc:507] Master 127.27.255.126:42717 requested a full tablet report, sending...
I20260505 20:18:00.983918 28669 webserver.cc:492] Webserver started at http://127.27.255.66:46659/ using document root <none> and password file <none>
I20260505 20:18:00.984484 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:00.984728 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:00.985061 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:00.985970 29012 ts_manager.cc:194] Registered new tserver with Master: 19b5c4a3f26e40419ed87526f4873650 (127.27.255.65:33599)
I20260505 20:18:00.986488 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/ts-1-root/instance:
uuid: "7715ff09d65c4399b50d8eea6e002b2c"
format_stamp: "Formatted at 2026-05-05 20:18:00 on dist-test-slave-dnnj"
I20260505 20:18:00.987967 29012 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56224
I20260505 20:18:00.991822 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260505 20:18:00.996057 29151 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:00.997179 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:00.997573 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/ts-1-root
uuid: "7715ff09d65c4399b50d8eea6e002b2c"
format_stamp: "Formatted at 2026-05-05 20:18:00 on dist-test-slave-dnnj"
I20260505 20:18:00.997907 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1778012272869005-28669-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:01.018251 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:01.019487 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:01.021047 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:01.023691 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:01.023918 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:01.024291 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:01.024448 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:01.075937 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.66:34059
I20260505 20:18:01.076009 29213 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.66:34059 every 8 connection(s)
I20260505 20:18:01.091033 29139 heartbeater.cc:499] Master 127.27.255.126:42717 was elected leader, sending a full tablet report...
I20260505 20:18:01.094698 29214 heartbeater.cc:344] Connected to a master server at 127.27.255.126:42717
I20260505 20:18:01.095113 29214 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:01.096100 29214 heartbeater.cc:507] Master 127.27.255.126:42717 requested a full tablet report, sending...
I20260505 20:18:01.099381 29012 ts_manager.cc:194] Registered new tserver with Master: 7715ff09d65c4399b50d8eea6e002b2c (127.27.255.66:34059)
I20260505 20:18:01.100206 28669 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.020374897s
I20260505 20:18:01.101475 29012 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56226
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:422: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20260505 20:18:01.128690 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:01.156172 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:01.174185 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:01.183223 28669 tablet_server.cc:179] TabletServer@127.27.255.66:0 shutting down...
I20260505 20:18:01.204978 29214 heartbeater.cc:499] Master 127.27.255.126:42717 was elected leader, sending a full tablet report...
I20260505 20:18:01.209473 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:01.226732 28669 tablet_server.cc:196] TabletServer@127.27.255.66:0 shutdown complete.
I20260505 20:18:01.235117 28669 master.cc:562] Master@127.27.255.126:42717 shutting down...
I20260505 20:18:01.253778 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:01.254352 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:01.254809 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P 27e8ee6206614f1dbf02327792bf6ee2: stopping tablet replica
I20260505 20:18:01.274539 28669 master.cc:584] Master@127.27.255.126:42717 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (666 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20260505 20:18:01.303941 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:39167
I20260505 20:18:01.305143 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:01.311010 29227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:01.312462 29228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:01.314128 29230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:01.314636 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:01.315681 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:01.315881 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:01.316016 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012281315986 us; error 0 us; skew 500 ppm
I20260505 20:18:01.316612 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:01.319780 28669 webserver.cc:492] Webserver started at http://127.27.255.126:38921/ using document root <none> and password file <none>
I20260505 20:18:01.320322 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:01.320502 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:01.320734 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:01.322068 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "31e52a050e304155be3fca7ca3ba0c39"
format_stamp: "Formatted at 2026-05-05 20:18:01 on dist-test-slave-dnnj"
I20260505 20:18:01.327069 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260505 20:18:01.331679 29235 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:01.332599 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260505 20:18:01.332984 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "31e52a050e304155be3fca7ca3ba0c39"
format_stamp: "Formatted at 2026-05-05 20:18:01 on dist-test-slave-dnnj"
I20260505 20:18:01.333348 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:01.351495 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:01.353170 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:01.408519 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:39167
I20260505 20:18:01.408695 29286 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:39167 every 8 connection(s)
I20260505 20:18:01.414469 29287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:01.427341 29287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39: Bootstrap starting.
I20260505 20:18:01.434895 29287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:01.440258 29287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39: No bootstrap required, opened a new log
I20260505 20:18:01.443048 29287 raft_consensus.cc:359] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e52a050e304155be3fca7ca3ba0c39" member_type: VOTER }
I20260505 20:18:01.443610 29287 raft_consensus.cc:385] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:01.443974 29287 raft_consensus.cc:740] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31e52a050e304155be3fca7ca3ba0c39, State: Initialized, Role: FOLLOWER
I20260505 20:18:01.444731 29287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e52a050e304155be3fca7ca3ba0c39" member_type: VOTER }
I20260505 20:18:01.445263 29287 raft_consensus.cc:399] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:01.445678 29287 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:01.445989 29287 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:01.453258 29287 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e52a050e304155be3fca7ca3ba0c39" member_type: VOTER }
I20260505 20:18:01.454125 29287 leader_election.cc:304] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31e52a050e304155be3fca7ca3ba0c39; no voters:
I20260505 20:18:01.455703 29287 leader_election.cc:290] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:01.456094 29290 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:01.458026 29290 raft_consensus.cc:697] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 1 LEADER]: Becoming Leader. State: Replica: 31e52a050e304155be3fca7ca3ba0c39, State: Running, Role: LEADER
I20260505 20:18:01.458869 29290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e52a050e304155be3fca7ca3ba0c39" member_type: VOTER }
I20260505 20:18:01.460549 29287 sys_catalog.cc:565] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:01.462199 29292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31e52a050e304155be3fca7ca3ba0c39. Latest consensus state: current_term: 1 leader_uuid: "31e52a050e304155be3fca7ca3ba0c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e52a050e304155be3fca7ca3ba0c39" member_type: VOTER } }
I20260505 20:18:01.462183 29291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31e52a050e304155be3fca7ca3ba0c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e52a050e304155be3fca7ca3ba0c39" member_type: VOTER } }
I20260505 20:18:01.462937 29292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:01.463114 29291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:01.467388 29294 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:01.477739 29294 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:01.482147 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:01.490264 29294 catalog_manager.cc:1357] Generated new cluster ID: b42b9e2a80394132bf778248f51ba9c2
I20260505 20:18:01.490779 29294 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:01.520543 29294 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:01.522604 29294 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:01.540908 29294 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39: Generated new TSK 0
I20260505 20:18:01.541906 29294 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:01.551059 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:01.562815 29308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:01.564982 29309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:01.566226 29311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:01.569728 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:01.570693 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:01.570914 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:01.571084 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012281571067 us; error 0 us; skew 500 ppm
I20260505 20:18:01.572036 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:01.576591 28669 webserver.cc:492] Webserver started at http://127.27.255.65:34649/ using document root <none> and password file <none>
I20260505 20:18:01.577206 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:01.577697 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:01.578066 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:01.579510 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "2e40b7a308d34d719590eb2d024faa24"
format_stamp: "Formatted at 2026-05-05 20:18:01 on dist-test-slave-dnnj"
I20260505 20:18:01.586004 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260505 20:18:01.590631 29316 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:01.591961 28669 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260505 20:18:01.592360 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "2e40b7a308d34d719590eb2d024faa24"
format_stamp: "Formatted at 2026-05-05 20:18:01 on dist-test-slave-dnnj"
I20260505 20:18:01.592720 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:01.614300 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:01.615691 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:01.617938 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:01.621222 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:01.621564 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:01.621857 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:01.622045 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:01.672788 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:36273
I20260505 20:18:01.672868 29378 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:36273 every 8 connection(s)
I20260505 20:18:01.678814 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:01.687114 29384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:01.688925 29383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:01.692111 29386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:01.692410 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:01.693410 29379 heartbeater.cc:344] Connected to a master server at 127.27.255.126:39167
I20260505 20:18:01.693815 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260505 20:18:01.693904 29379 heartbeater.cc:461] Registering TS with master...
W20260505 20:18:01.694078 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:01.694322 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012281694300 us; error 0 us; skew 500 ppm
I20260505 20:18:01.695004 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:01.695031 29379 heartbeater.cc:507] Master 127.27.255.126:39167 requested a full tablet report, sending...
I20260505 20:18:01.697988 29252 ts_manager.cc:194] Registered new tserver with Master: 2e40b7a308d34d719590eb2d024faa24 (127.27.255.65:36273)
I20260505 20:18:01.698403 28669 webserver.cc:492] Webserver started at http://127.27.255.66:42987/ using document root <none> and password file <none>
I20260505 20:18:01.698984 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:01.699236 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:01.699546 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:01.701152 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/ts-1-root/instance:
uuid: "4722c11c2120491abf9a75efdadc0884"
format_stamp: "Formatted at 2026-05-05 20:18:01 on dist-test-slave-dnnj"
I20260505 20:18:01.701290 29252 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52792
I20260505 20:18:01.707576 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260505 20:18:01.712013 29391 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:01.713459 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:01.713950 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/ts-1-root
uuid: "4722c11c2120491abf9a75efdadc0884"
format_stamp: "Formatted at 2026-05-05 20:18:01 on dist-test-slave-dnnj"
I20260505 20:18:01.714304 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1778012272869005-28669-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:01.732254 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:01.733345 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:01.735042 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:01.737797 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:01.738003 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:01.738375 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:01.738708 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:01.790475 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.66:37189
I20260505 20:18:01.790613 29453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.66:37189 every 8 connection(s)
I20260505 20:18:01.804901 29379 heartbeater.cc:499] Master 127.27.255.126:39167 was elected leader, sending a full tablet report...
I20260505 20:18:01.806278 29454 heartbeater.cc:344] Connected to a master server at 127.27.255.126:39167
I20260505 20:18:01.806742 29454 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:01.807632 29454 heartbeater.cc:507] Master 127.27.255.126:39167 requested a full tablet report, sending...
I20260505 20:18:01.809885 29252 ts_manager.cc:194] Registered new tserver with Master: 4722c11c2120491abf9a75efdadc0884 (127.27.255.66:37189)
I20260505 20:18:01.810029 28669 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.016417099s
I20260505 20:18:01.811580 29252 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52798
I20260505 20:18:01.839107 29252 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260505 20:18:01.840746 29252 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260505 20:18:01.886366 29419 tablet_service.cc:1511] Processing CreateTablet for tablet 297470b1ea0a40fd84565bc54f03ca9b (DEFAULT_TABLE table=test-table [id=3716cd773df34274b1744e4c2699c920]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:01.887989 29419 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297470b1ea0a40fd84565bc54f03ca9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:01.890832 29344 tablet_service.cc:1511] Processing CreateTablet for tablet 297470b1ea0a40fd84565bc54f03ca9b (DEFAULT_TABLE table=test-table [id=3716cd773df34274b1744e4c2699c920]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:01.892035 29344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297470b1ea0a40fd84565bc54f03ca9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:01.909065 29467 tablet_bootstrap.cc:492] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884: Bootstrap starting.
I20260505 20:18:01.913046 29468 tablet_bootstrap.cc:492] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24: Bootstrap starting.
I20260505 20:18:01.914510 29454 heartbeater.cc:499] Master 127.27.255.126:39167 was elected leader, sending a full tablet report...
I20260505 20:18:01.916054 29467 tablet_bootstrap.cc:654] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:01.919463 29468 tablet_bootstrap.cc:654] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:01.924911 29467 tablet_bootstrap.cc:492] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884: No bootstrap required, opened a new log
I20260505 20:18:01.925405 29467 ts_tablet_manager.cc:1403] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260505 20:18:01.927932 29467 raft_consensus.cc:359] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 } } peers { permanent_uuid: "2e40b7a308d34d719590eb2d024faa24" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36273 } }
I20260505 20:18:01.928443 29467 raft_consensus.cc:740] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4722c11c2120491abf9a75efdadc0884, State: Initialized, Role: FOLLOWER
I20260505 20:18:01.928957 29468 tablet_bootstrap.cc:492] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24: No bootstrap required, opened a new log
I20260505 20:18:01.929754 29468 ts_tablet_manager.cc:1403] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260505 20:18:01.929658 29467 consensus_queue.cc:260] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 } } peers { permanent_uuid: "2e40b7a308d34d719590eb2d024faa24" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36273 } }
I20260505 20:18:01.932811 29468 raft_consensus.cc:359] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 } } peers { permanent_uuid: "2e40b7a308d34d719590eb2d024faa24" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36273 } }
I20260505 20:18:01.933439 29468 raft_consensus.cc:740] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e40b7a308d34d719590eb2d024faa24, State: Initialized, Role: FOLLOWER
I20260505 20:18:01.933476 29467 ts_tablet_manager.cc:1434] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260505 20:18:01.934098 29468 consensus_queue.cc:260] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 } } peers { permanent_uuid: "2e40b7a308d34d719590eb2d024faa24" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36273 } }
I20260505 20:18:01.938374 29468 ts_tablet_manager.cc:1434] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24: Time spent starting tablet: real 0.008s user 0.001s sys 0.007s
I20260505 20:18:01.973186 28669 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260505 20:18:01.973394 28669 raft_consensus.cc:423] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 0 FOLLOWER]: Emulating election...
I20260505 20:18:01.973747 28669 raft_consensus.cc:3060] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:01.977770 28669 raft_consensus.cc:697] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 1 LEADER]: Becoming Leader. State: Replica: 2e40b7a308d34d719590eb2d024faa24, State: Running, Role: LEADER
W20260505 20:18:01.978510 29379 heartbeater.cc:646] Failed to heartbeat to 127.27.255.126:39167 (0 consecutive failures): IO error: failing all heartbeats for tests
I20260505 20:18:01.978956 28669 consensus_queue.cc:237] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 } } peers { permanent_uuid: "2e40b7a308d34d719590eb2d024faa24" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36273 } }
I20260505 20:18:01.981709 28669 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260505 20:18:02.002422 28669 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260505 20:18:02.108173 28669 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260505 20:18:02.312915 28669 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260505 20:18:02.423934 29429 raft_consensus.cc:3060] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:02.430950 29472 consensus_queue.cc:1048] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260505 20:18:02.618974 28669 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260505 20:18:02.619199 28669 raft_consensus.cc:423] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 1 LEADER]: Emulating election...
I20260505 20:18:02.619484 28669 raft_consensus.cc:3055] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 1 LEADER]: Stepping down as leader of term 1
I20260505 20:18:02.619724 28669 raft_consensus.cc:740] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2e40b7a308d34d719590eb2d024faa24, State: Running, Role: LEADER
I20260505 20:18:02.620220 28669 consensus_queue.cc:260] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 } } peers { permanent_uuid: "2e40b7a308d34d719590eb2d024faa24" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36273 } }
I20260505 20:18:02.620891 28669 raft_consensus.cc:3060] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:02.624866 28669 raft_consensus.cc:697] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 2 LEADER]: Becoming Leader. State: Replica: 2e40b7a308d34d719590eb2d024faa24, State: Running, Role: LEADER
I20260505 20:18:02.625424 28669 consensus_queue.cc:237] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 } } peers { permanent_uuid: "2e40b7a308d34d719590eb2d024faa24" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36273 } }
I20260505 20:18:02.626837 28669 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260505 20:18:02.630887 28669 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260505 20:18:02.736603 28669 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260505 20:18:02.942600 28669 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260505 20:18:03.206171 29429 raft_consensus.cc:3060] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:03.213338 29429 raft_consensus.cc:1275] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884 [term 2 FOLLOWER]: Refusing update from remote peer 2e40b7a308d34d719590eb2d024faa24: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260505 20:18:03.215483 29482 consensus_queue.cc:1048] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260505 20:18:03.250962 28669 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260505 20:18:03.251241 28669 raft_consensus.cc:423] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 2 LEADER]: Emulating election...
I20260505 20:18:03.251468 28669 raft_consensus.cc:3055] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 2 LEADER]: Stepping down as leader of term 2
I20260505 20:18:03.251660 28669 raft_consensus.cc:740] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2e40b7a308d34d719590eb2d024faa24, State: Running, Role: LEADER
I20260505 20:18:03.252136 28669 consensus_queue.cc:260] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 } } peers { permanent_uuid: "2e40b7a308d34d719590eb2d024faa24" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36273 } }
I20260505 20:18:03.252920 28669 raft_consensus.cc:3060] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:18:03.258787 28669 raft_consensus.cc:697] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 3 LEADER]: Becoming Leader. State: Replica: 2e40b7a308d34d719590eb2d024faa24, State: Running, Role: LEADER
I20260505 20:18:03.259617 28669 consensus_queue.cc:237] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 } } peers { permanent_uuid: "2e40b7a308d34d719590eb2d024faa24" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36273 } }
I20260505 20:18:03.261480 28669 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260505 20:18:03.266461 28669 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260505 20:18:03.371769 28669 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260505 20:18:03.578043 28669 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260505 20:18:03.711686 29429 raft_consensus.cc:3060] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884 [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:18:03.717736 29429 raft_consensus.cc:1275] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884 [term 3 FOLLOWER]: Refusing update from remote peer 2e40b7a308d34d719590eb2d024faa24: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260505 20:18:03.719401 29482 consensus_queue.cc:1048] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4722c11c2120491abf9a75efdadc0884" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260505 20:18:03.888029 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:03.925382 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:03.926717 28669 tablet_replica.cc:333] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24: stopping tablet replica
I20260505 20:18:03.927439 28669 raft_consensus.cc:2243] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 3 LEADER]: Raft consensus shutting down.
I20260505 20:18:03.928194 28669 raft_consensus.cc:2272] T 297470b1ea0a40fd84565bc54f03ca9b P 2e40b7a308d34d719590eb2d024faa24 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:03.951092 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:03.963372 28669 tablet_server.cc:179] TabletServer@127.27.255.66:0 shutting down...
I20260505 20:18:03.988840 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:03.989825 28669 tablet_replica.cc:333] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884: stopping tablet replica
I20260505 20:18:03.990474 28669 raft_consensus.cc:2243] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:03.991025 28669 raft_consensus.cc:2272] T 297470b1ea0a40fd84565bc54f03ca9b P 4722c11c2120491abf9a75efdadc0884 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:04.004074 28669 tablet_server.cc:196] TabletServer@127.27.255.66:0 shutdown complete.
I20260505 20:18:04.015252 28669 master.cc:562] Master@127.27.255.126:39167 shutting down...
I20260505 20:18:04.039323 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:04.040100 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:04.040429 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P 31e52a050e304155be3fca7ca3ba0c39: stopping tablet replica
I20260505 20:18:04.061048 28669 master.cc:584] Master@127.27.255.126:39167 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2795 ms)
[----------] 3 tests from LeadershipChangeReportingTest (11054 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20260505 20:18:04.099563 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:37917
I20260505 20:18:04.100942 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:04.108196 29493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:04.108871 29494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:04.110704 29496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:04.111615 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:04.112432 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:04.112663 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:04.112788 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012284112776 us; error 0 us; skew 500 ppm
I20260505 20:18:04.113286 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:04.116416 28669 webserver.cc:492] Webserver started at http://127.27.255.126:40643/ using document root <none> and password file <none>
I20260505 20:18:04.116931 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:04.117110 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:04.117365 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:04.118782 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "4d8ed1f561cf43729cbbfc50d2346d79"
format_stamp: "Formatted at 2026-05-05 20:18:04 on dist-test-slave-dnnj"
I20260505 20:18:04.123804 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260505 20:18:04.127708 29501 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:04.128710 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:04.129069 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "4d8ed1f561cf43729cbbfc50d2346d79"
format_stamp: "Formatted at 2026-05-05 20:18:04 on dist-test-slave-dnnj"
I20260505 20:18:04.129413 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:04.147049 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:04.148568 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:04.195484 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:37917
I20260505 20:18:04.195617 29552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:37917 every 8 connection(s)
I20260505 20:18:04.200132 29553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:04.213483 29553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79: Bootstrap starting.
I20260505 20:18:04.219017 29553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:04.224514 29553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79: No bootstrap required, opened a new log
I20260505 20:18:04.227210 29553 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8ed1f561cf43729cbbfc50d2346d79" member_type: VOTER }
I20260505 20:18:04.227682 29553 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:04.227977 29553 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8ed1f561cf43729cbbfc50d2346d79, State: Initialized, Role: FOLLOWER
I20260505 20:18:04.228628 29553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8ed1f561cf43729cbbfc50d2346d79" member_type: VOTER }
I20260505 20:18:04.229118 29553 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:04.229385 29553 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:04.229704 29553 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:04.235260 29553 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8ed1f561cf43729cbbfc50d2346d79" member_type: VOTER }
I20260505 20:18:04.235931 29553 leader_election.cc:304] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d8ed1f561cf43729cbbfc50d2346d79; no voters:
I20260505 20:18:04.237288 29553 leader_election.cc:290] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:04.237967 29556 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:04.239434 29556 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8ed1f561cf43729cbbfc50d2346d79, State: Running, Role: LEADER
I20260505 20:18:04.240190 29556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8ed1f561cf43729cbbfc50d2346d79" member_type: VOTER }
I20260505 20:18:04.240804 29553 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:04.243640 29558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d8ed1f561cf43729cbbfc50d2346d79. Latest consensus state: current_term: 1 leader_uuid: "4d8ed1f561cf43729cbbfc50d2346d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8ed1f561cf43729cbbfc50d2346d79" member_type: VOTER } }
I20260505 20:18:04.243746 29557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d8ed1f561cf43729cbbfc50d2346d79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8ed1f561cf43729cbbfc50d2346d79" member_type: VOTER } }
I20260505 20:18:04.244369 29558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:04.244653 29557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:04.251946 29561 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:04.258188 29561 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:04.260648 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:04.268599 29561 catalog_manager.cc:1357] Generated new cluster ID: f37a053e68ed422d87375c0c6233c2c7
I20260505 20:18:04.268980 29561 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:04.293653 29561 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:04.295185 29561 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:04.314580 29561 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79: Generated new TSK 0
I20260505 20:18:04.315308 29561 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:04.328063 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:04.336061 29574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:04.336882 29575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:04.338958 29577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:04.339936 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:04.340909 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:04.341118 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:04.341338 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012284341325 us; error 0 us; skew 500 ppm
I20260505 20:18:04.342005 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:04.344955 28669 webserver.cc:492] Webserver started at http://127.27.255.65:39981/ using document root <none> and password file <none>
I20260505 20:18:04.345657 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:04.345862 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:04.346177 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:04.348002 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "791033b1c7df4ee6ad7aad6f46c46395"
format_stamp: "Formatted at 2026-05-05 20:18:04 on dist-test-slave-dnnj"
I20260505 20:18:04.354324 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260505 20:18:04.358629 29582 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:04.359634 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260505 20:18:04.359905 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "791033b1c7df4ee6ad7aad6f46c46395"
format_stamp: "Formatted at 2026-05-05 20:18:04 on dist-test-slave-dnnj"
I20260505 20:18:04.360176 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:04.371282 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:04.372802 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:04.374796 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:04.377565 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:04.377785 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:04.377975 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:04.378126 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:04.431329 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:44705
I20260505 20:18:04.431447 29644 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:44705 every 8 connection(s)
I20260505 20:18:04.437556 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:04.446954 29649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:04.447497 29650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:04.454201 29652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:04.455444 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:04.456035 29645 heartbeater.cc:344] Connected to a master server at 127.27.255.126:37917
I20260505 20:18:04.456516 29645 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:04.456800 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:04.457056 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:04.457242 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012284457222 us; error 0 us; skew 500 ppm
I20260505 20:18:04.457448 29645 heartbeater.cc:507] Master 127.27.255.126:37917 requested a full tablet report, sending...
I20260505 20:18:04.457964 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:04.460472 29518 ts_manager.cc:194] Registered new tserver with Master: 791033b1c7df4ee6ad7aad6f46c46395 (127.27.255.65:44705)
I20260505 20:18:04.464052 29518 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53102
I20260505 20:18:04.465214 28669 webserver.cc:492] Webserver started at http://127.27.255.66:37707/ using document root <none> and password file <none>
I20260505 20:18:04.465924 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:04.466172 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:04.466547 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:04.468148 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-1-root/instance:
uuid: "79c8028dadf2453b8b52d106a0f02618"
format_stamp: "Formatted at 2026-05-05 20:18:04 on dist-test-slave-dnnj"
I20260505 20:18:04.473172 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260505 20:18:04.477164 29657 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:04.478145 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260505 20:18:04.478600 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-1-root
uuid: "79c8028dadf2453b8b52d106a0f02618"
format_stamp: "Formatted at 2026-05-05 20:18:04 on dist-test-slave-dnnj"
I20260505 20:18:04.478904 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:04.501008 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:04.502633 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:04.504711 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:04.507488 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:04.507758 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:04.508069 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:04.508273 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:04.560967 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.66:44109
I20260505 20:18:04.561072 29719 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.66:44109 every 8 connection(s)
I20260505 20:18:04.566787 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:04.576169 29723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:04.578191 29724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:04.582212 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:04.583114 29720 heartbeater.cc:344] Connected to a master server at 127.27.255.126:37917
I20260505 20:18:04.583748 29720 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:04.584882 29720 heartbeater.cc:507] Master 127.27.255.126:37917 requested a full tablet report, sending...
W20260505 20:18:04.584887 29726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:04.586253 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:04.586584 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:04.586839 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012284586793 us; error 0 us; skew 500 ppm
I20260505 20:18:04.587623 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:04.587809 29518 ts_manager.cc:194] Registered new tserver with Master: 79c8028dadf2453b8b52d106a0f02618 (127.27.255.66:44109)
I20260505 20:18:04.590075 29518 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53108
I20260505 20:18:04.591194 28669 webserver.cc:492] Webserver started at http://127.27.255.67:33499/ using document root <none> and password file <none>
I20260505 20:18:04.591958 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:04.592196 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:04.592545 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:04.594072 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-2-root/instance:
uuid: "66336f70a8ad4a42b8722ee5c5e1aed0"
format_stamp: "Formatted at 2026-05-05 20:18:04 on dist-test-slave-dnnj"
I20260505 20:18:04.599888 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260505 20:18:04.604094 29731 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:04.605154 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:04.605881 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-2-root
uuid: "66336f70a8ad4a42b8722ee5c5e1aed0"
format_stamp: "Formatted at 2026-05-05 20:18:04 on dist-test-slave-dnnj"
I20260505 20:18:04.606242 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1778012272869005-28669-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:04.620874 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:04.622411 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:04.624122 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:04.626974 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:04.627264 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:04.627526 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:04.627715 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:04.718542 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.67:39163
I20260505 20:18:04.718667 29793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.67:39163 every 8 connection(s)
I20260505 20:18:04.738610 29794 heartbeater.cc:344] Connected to a master server at 127.27.255.126:37917
I20260505 20:18:04.739486 29794 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:04.740621 29794 heartbeater.cc:507] Master 127.27.255.126:37917 requested a full tablet report, sending...
I20260505 20:18:04.743444 29518 ts_manager.cc:194] Registered new tserver with Master: 66336f70a8ad4a42b8722ee5c5e1aed0 (127.27.255.67:39163)
I20260505 20:18:04.743539 28669 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019498565s
I20260505 20:18:04.745010 29518 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53114
I20260505 20:18:04.775486 29518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260505 20:18:04.777410 29518 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260505 20:18:04.821214 29610 tablet_service.cc:1511] Processing CreateTablet for tablet d0d4af2c34d44e7d958018c70bd3bc7e (DEFAULT_TABLE table=test-table [id=e7af36512555437dacfdddf74a3b17fd]), partition=
I20260505 20:18:04.822472 29610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0d4af2c34d44e7d958018c70bd3bc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:04.822494 29759 tablet_service.cc:1511] Processing CreateTablet for tablet d0d4af2c34d44e7d958018c70bd3bc7e (DEFAULT_TABLE table=test-table [id=e7af36512555437dacfdddf74a3b17fd]), partition=
I20260505 20:18:04.823632 29759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0d4af2c34d44e7d958018c70bd3bc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:04.824187 29685 tablet_service.cc:1511] Processing CreateTablet for tablet d0d4af2c34d44e7d958018c70bd3bc7e (DEFAULT_TABLE table=test-table [id=e7af36512555437dacfdddf74a3b17fd]), partition=
I20260505 20:18:04.825358 29685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0d4af2c34d44e7d958018c70bd3bc7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:04.838089 29809 tablet_bootstrap.cc:492] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618: Bootstrap starting.
I20260505 20:18:04.841895 29810 tablet_bootstrap.cc:492] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395: Bootstrap starting.
I20260505 20:18:04.843752 29811 tablet_bootstrap.cc:492] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0: Bootstrap starting.
I20260505 20:18:04.847886 29809 tablet_bootstrap.cc:654] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:04.849293 29811 tablet_bootstrap.cc:654] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:04.849843 29810 tablet_bootstrap.cc:654] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:04.860069 29809 tablet_bootstrap.cc:492] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618: No bootstrap required, opened a new log
I20260505 20:18:04.860553 29809 ts_tablet_manager.cc:1403] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.007s
I20260505 20:18:04.861001 29811 tablet_bootstrap.cc:492] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0: No bootstrap required, opened a new log
I20260505 20:18:04.861608 29811 ts_tablet_manager.cc:1403] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20260505 20:18:04.862167 29810 tablet_bootstrap.cc:492] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395: No bootstrap required, opened a new log
I20260505 20:18:04.862627 29810 ts_tablet_manager.cc:1403] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.006s
I20260505 20:18:04.863729 29809 raft_consensus.cc:359] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:04.864035 29811 raft_consensus.cc:359] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:04.865257 29811 raft_consensus.cc:385] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:04.864598 29809 raft_consensus.cc:385] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:04.865598 29811 raft_consensus.cc:740] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66336f70a8ad4a42b8722ee5c5e1aed0, State: Initialized, Role: FOLLOWER
I20260505 20:18:04.865931 29809 raft_consensus.cc:740] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c8028dadf2453b8b52d106a0f02618, State: Initialized, Role: FOLLOWER
I20260505 20:18:04.866360 29810 raft_consensus.cc:359] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:04.866596 29811 consensus_queue.cc:260] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:04.867116 29810 raft_consensus.cc:385] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:04.866755 29809 consensus_queue.cc:260] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:04.867483 29810 raft_consensus.cc:740] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 791033b1c7df4ee6ad7aad6f46c46395, State: Initialized, Role: FOLLOWER
I20260505 20:18:04.869343 29810 consensus_queue.cc:260] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:04.870163 29794 heartbeater.cc:499] Master 127.27.255.126:37917 was elected leader, sending a full tablet report...
I20260505 20:18:04.871194 29720 heartbeater.cc:499] Master 127.27.255.126:37917 was elected leader, sending a full tablet report...
I20260505 20:18:04.871322 29811 ts_tablet_manager.cc:1434] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260505 20:18:04.873292 29645 heartbeater.cc:499] Master 127.27.255.126:37917 was elected leader, sending a full tablet report...
I20260505 20:18:04.873355 29809 ts_tablet_manager.cc:1434] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20260505 20:18:04.873864 29810 ts_tablet_manager.cc:1434] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260505 20:18:05.117617 29817 raft_consensus.cc:493] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:05.118124 29817 raft_consensus.cc:515] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:05.120222 29817 leader_election.cc:290] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66336f70a8ad4a42b8722ee5c5e1aed0 (127.27.255.67:39163), 79c8028dadf2453b8b52d106a0f02618 (127.27.255.66:44109)
I20260505 20:18:05.132602 29769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0d4af2c34d44e7d958018c70bd3bc7e" candidate_uuid: "791033b1c7df4ee6ad7aad6f46c46395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" is_pre_election: true
I20260505 20:18:05.132717 29695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0d4af2c34d44e7d958018c70bd3bc7e" candidate_uuid: "791033b1c7df4ee6ad7aad6f46c46395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79c8028dadf2453b8b52d106a0f02618" is_pre_election: true
I20260505 20:18:05.133513 29769 raft_consensus.cc:2468] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791033b1c7df4ee6ad7aad6f46c46395 in term 0.
I20260505 20:18:05.133693 29695 raft_consensus.cc:2468] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 791033b1c7df4ee6ad7aad6f46c46395 in term 0.
I20260505 20:18:05.135046 29583 leader_election.cc:304] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66336f70a8ad4a42b8722ee5c5e1aed0, 791033b1c7df4ee6ad7aad6f46c46395; no voters:
I20260505 20:18:05.135888 29817 raft_consensus.cc:2804] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260505 20:18:05.136279 29817 raft_consensus.cc:493] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:05.136559 29817 raft_consensus.cc:3060] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:05.144407 29817 raft_consensus.cc:515] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:05.146281 29817 leader_election.cc:290] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [CANDIDATE]: Term 1 election: Requested vote from peers 66336f70a8ad4a42b8722ee5c5e1aed0 (127.27.255.67:39163), 79c8028dadf2453b8b52d106a0f02618 (127.27.255.66:44109)
I20260505 20:18:05.147334 29769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0d4af2c34d44e7d958018c70bd3bc7e" candidate_uuid: "791033b1c7df4ee6ad7aad6f46c46395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0"
I20260505 20:18:05.147392 29695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0d4af2c34d44e7d958018c70bd3bc7e" candidate_uuid: "791033b1c7df4ee6ad7aad6f46c46395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79c8028dadf2453b8b52d106a0f02618"
I20260505 20:18:05.147887 29769 raft_consensus.cc:3060] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:05.147943 29695 raft_consensus.cc:3060] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:05.153136 29769 raft_consensus.cc:2468] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791033b1c7df4ee6ad7aad6f46c46395 in term 1.
I20260505 20:18:05.153923 29695 raft_consensus.cc:2468] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 791033b1c7df4ee6ad7aad6f46c46395 in term 1.
I20260505 20:18:05.154763 29583 leader_election.cc:304] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66336f70a8ad4a42b8722ee5c5e1aed0, 791033b1c7df4ee6ad7aad6f46c46395; no voters:
I20260505 20:18:05.155747 29817 raft_consensus.cc:2804] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:05.157024 29817 raft_consensus.cc:697] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 1 LEADER]: Becoming Leader. State: Replica: 791033b1c7df4ee6ad7aad6f46c46395, State: Running, Role: LEADER
I20260505 20:18:05.157824 29817 consensus_queue.cc:237] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:05.164837 29518 catalog_manager.cc:5671] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 reported cstate change: term changed from 0 to 1, leader changed from <none> to 791033b1c7df4ee6ad7aad6f46c46395 (127.27.255.65). New cstate: current_term: 1 leader_uuid: "791033b1c7df4ee6ad7aad6f46c46395" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } health_report { overall_health: UNKNOWN } } }
I20260505 20:18:05.538465 29817 consensus_queue.cc:1048] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260505 20:18:05.552614 29826 consensus_queue.cc:1048] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260505 20:18:06.253382 28669 tablet_replica.cc:333] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395: stopping tablet replica
I20260505 20:18:06.254143 28669 raft_consensus.cc:2243] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:06.255064 28669 raft_consensus.cc:2272] T d0d4af2c34d44e7d958018c70bd3bc7e P 791033b1c7df4ee6ad7aad6f46c46395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:07.515205 29831 raft_consensus.cc:493] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 791033b1c7df4ee6ad7aad6f46c46395)
I20260505 20:18:07.515843 29831 raft_consensus.cc:515] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:07.518059 29831 leader_election.cc:290] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 791033b1c7df4ee6ad7aad6f46c46395 (127.27.255.65:44705), 66336f70a8ad4a42b8722ee5c5e1aed0 (127.27.255.67:39163)
I20260505 20:18:07.532625 29620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0d4af2c34d44e7d958018c70bd3bc7e" candidate_uuid: "79c8028dadf2453b8b52d106a0f02618" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "791033b1c7df4ee6ad7aad6f46c46395" is_pre_election: true
I20260505 20:18:07.533771 29769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0d4af2c34d44e7d958018c70bd3bc7e" candidate_uuid: "79c8028dadf2453b8b52d106a0f02618" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" is_pre_election: true
W20260505 20:18:07.534504 29658 leader_election.cc:343] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 791033b1c7df4ee6ad7aad6f46c46395 (127.27.255.65:44705): Illegal state: cannot vote while shut down
I20260505 20:18:07.535598 29658 leader_election.cc:304] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79c8028dadf2453b8b52d106a0f02618; no voters: 66336f70a8ad4a42b8722ee5c5e1aed0, 791033b1c7df4ee6ad7aad6f46c46395
I20260505 20:18:07.536429 29831 raft_consensus.cc:2749] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260505 20:18:07.551036 29836 raft_consensus.cc:493] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 791033b1c7df4ee6ad7aad6f46c46395)
I20260505 20:18:07.551767 29836 raft_consensus.cc:515] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:07.554100 29836 leader_election.cc:290] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 791033b1c7df4ee6ad7aad6f46c46395 (127.27.255.65:44705), 79c8028dadf2453b8b52d106a0f02618 (127.27.255.66:44109)
I20260505 20:18:07.567996 29620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0d4af2c34d44e7d958018c70bd3bc7e" candidate_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "791033b1c7df4ee6ad7aad6f46c46395" is_pre_election: true
I20260505 20:18:07.568506 29695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0d4af2c34d44e7d958018c70bd3bc7e" candidate_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "79c8028dadf2453b8b52d106a0f02618" is_pre_election: true
I20260505 20:18:07.569104 29695 raft_consensus.cc:2468] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66336f70a8ad4a42b8722ee5c5e1aed0 in term 1.
W20260505 20:18:07.569327 29732 leader_election.cc:343] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 791033b1c7df4ee6ad7aad6f46c46395 (127.27.255.65:44705): Illegal state: cannot vote while shut down
I20260505 20:18:07.570421 29733 leader_election.cc:304] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66336f70a8ad4a42b8722ee5c5e1aed0, 79c8028dadf2453b8b52d106a0f02618; no voters: 791033b1c7df4ee6ad7aad6f46c46395
I20260505 20:18:07.571200 29836 raft_consensus.cc:2804] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260505 20:18:07.571540 29836 raft_consensus.cc:493] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 791033b1c7df4ee6ad7aad6f46c46395)
I20260505 20:18:07.571918 29836 raft_consensus.cc:3060] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:07.577936 29836 raft_consensus.cc:515] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:07.579838 29836 leader_election.cc:290] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [CANDIDATE]: Term 2 election: Requested vote from peers 791033b1c7df4ee6ad7aad6f46c46395 (127.27.255.65:44705), 79c8028dadf2453b8b52d106a0f02618 (127.27.255.66:44109)
I20260505 20:18:07.581118 29620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0d4af2c34d44e7d958018c70bd3bc7e" candidate_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "791033b1c7df4ee6ad7aad6f46c46395"
I20260505 20:18:07.581272 29695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0d4af2c34d44e7d958018c70bd3bc7e" candidate_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "79c8028dadf2453b8b52d106a0f02618"
I20260505 20:18:07.581960 29695 raft_consensus.cc:3060] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 1 FOLLOWER]: Advancing to term 2
W20260505 20:18:07.582454 29732 leader_election.cc:343] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 791033b1c7df4ee6ad7aad6f46c46395 (127.27.255.65:44705): Illegal state: cannot vote while shut down
I20260505 20:18:07.590440 29695 raft_consensus.cc:2468] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66336f70a8ad4a42b8722ee5c5e1aed0 in term 2.
I20260505 20:18:07.591691 29733 leader_election.cc:304] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66336f70a8ad4a42b8722ee5c5e1aed0, 79c8028dadf2453b8b52d106a0f02618; no voters: 791033b1c7df4ee6ad7aad6f46c46395
I20260505 20:18:07.592407 29836 raft_consensus.cc:2804] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 2 FOLLOWER]: Leader election won for term 2
I20260505 20:18:07.593746 29836 raft_consensus.cc:697] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 2 LEADER]: Becoming Leader. State: Replica: 66336f70a8ad4a42b8722ee5c5e1aed0, State: Running, Role: LEADER
I20260505 20:18:07.594538 29836 consensus_queue.cc:237] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "791033b1c7df4ee6ad7aad6f46c46395" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 44705 } } peers { permanent_uuid: "66336f70a8ad4a42b8722ee5c5e1aed0" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 39163 } } peers { permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 } }
I20260505 20:18:07.983883 29695 raft_consensus.cc:1275] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 2 FOLLOWER]: Refusing update from remote peer 66336f70a8ad4a42b8722ee5c5e1aed0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260505 20:18:07.985536 29836 consensus_queue.cc:1048] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79c8028dadf2453b8b52d106a0f02618" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260505 20:18:07.997098 29732 consensus_queue.cc:579] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [LEADER]: The tablet replica hosted on peer 791033b1c7df4ee6ad7aad6f46c46395 has failed
W20260505 20:18:07.997834 29732 consensus_peers.cc:597] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 -> Peer 791033b1c7df4ee6ad7aad6f46c46395 (127.27.255.65:44705): Couldn't send request to peer 791033b1c7df4ee6ad7aad6f46c46395. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260505 20:18:08.293823 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:08.316753 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:08.335533 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:08.346136 28669 tablet_server.cc:179] TabletServer@127.27.255.66:0 shutting down...
I20260505 20:18:08.374138 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:08.375412 28669 tablet_replica.cc:333] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618: stopping tablet replica
I20260505 20:18:08.376180 28669 raft_consensus.cc:2243] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:08.376763 28669 raft_consensus.cc:2272] T d0d4af2c34d44e7d958018c70bd3bc7e P 79c8028dadf2453b8b52d106a0f02618 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:08.400897 28669 tablet_server.cc:196] TabletServer@127.27.255.66:0 shutdown complete.
I20260505 20:18:08.411298 28669 tablet_server.cc:179] TabletServer@127.27.255.67:0 shutting down...
I20260505 20:18:08.431793 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:08.432446 28669 tablet_replica.cc:333] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0: stopping tablet replica
I20260505 20:18:08.432917 28669 raft_consensus.cc:2243] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 2 LEADER]: Raft consensus shutting down.
I20260505 20:18:08.433813 28669 raft_consensus.cc:2272] T d0d4af2c34d44e7d958018c70bd3bc7e P 66336f70a8ad4a42b8722ee5c5e1aed0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:08.455263 28669 tablet_server.cc:196] TabletServer@127.27.255.67:0 shutdown complete.
I20260505 20:18:08.464815 28669 master.cc:562] Master@127.27.255.126:37917 shutting down...
I20260505 20:18:08.482779 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:08.483464 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:08.483798 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4d8ed1f561cf43729cbbfc50d2346d79: stopping tablet replica
I20260505 20:18:08.503458 28669 master.cc:584] Master@127.27.255.126:37917 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (4434 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20260505 20:18:08.533910 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:43573
I20260505 20:18:08.535344 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:08.541347 29850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:08.542305 29851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:08.545019 29853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:08.545430 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:08.546379 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:08.546561 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:08.546679 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012288546667 us; error 0 us; skew 500 ppm
I20260505 20:18:08.547158 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:08.549851 28669 webserver.cc:492] Webserver started at http://127.27.255.126:35989/ using document root <none> and password file <none>
I20260505 20:18:08.550352 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:08.550518 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:08.550748 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:08.551919 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "75091d4322ed41f9a0b4f85844a63abd"
format_stamp: "Formatted at 2026-05-05 20:18:08 on dist-test-slave-dnnj"
I20260505 20:18:08.556833 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260505 20:18:08.561003 29858 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:08.561951 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260505 20:18:08.562341 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "75091d4322ed41f9a0b4f85844a63abd"
format_stamp: "Formatted at 2026-05-05 20:18:08 on dist-test-slave-dnnj"
I20260505 20:18:08.562690 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:08.579198 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:08.580483 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:08.626431 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:43573
I20260505 20:18:08.626560 29909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:43573 every 8 connection(s)
I20260505 20:18:08.630978 29910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:08.643013 29910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd: Bootstrap starting.
I20260505 20:18:08.648321 29910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:08.653324 29910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd: No bootstrap required, opened a new log
I20260505 20:18:08.656256 29910 raft_consensus.cc:359] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75091d4322ed41f9a0b4f85844a63abd" member_type: VOTER }
I20260505 20:18:08.656786 29910 raft_consensus.cc:385] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:08.656993 29910 raft_consensus.cc:740] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75091d4322ed41f9a0b4f85844a63abd, State: Initialized, Role: FOLLOWER
I20260505 20:18:08.657709 29910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75091d4322ed41f9a0b4f85844a63abd" member_type: VOTER }
I20260505 20:18:08.658207 29910 raft_consensus.cc:399] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:08.658420 29910 raft_consensus.cc:493] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:08.658674 29910 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:08.664755 29910 raft_consensus.cc:515] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75091d4322ed41f9a0b4f85844a63abd" member_type: VOTER }
I20260505 20:18:08.665441 29910 leader_election.cc:304] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75091d4322ed41f9a0b4f85844a63abd; no voters:
I20260505 20:18:08.666796 29910 leader_election.cc:290] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:08.667187 29913 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:08.668731 29913 raft_consensus.cc:697] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 1 LEADER]: Becoming Leader. State: Replica: 75091d4322ed41f9a0b4f85844a63abd, State: Running, Role: LEADER
I20260505 20:18:08.669483 29913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75091d4322ed41f9a0b4f85844a63abd" member_type: VOTER }
I20260505 20:18:08.670068 29910 sys_catalog.cc:565] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:08.672847 29915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75091d4322ed41f9a0b4f85844a63abd. Latest consensus state: current_term: 1 leader_uuid: "75091d4322ed41f9a0b4f85844a63abd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75091d4322ed41f9a0b4f85844a63abd" member_type: VOTER } }
I20260505 20:18:08.672807 29914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75091d4322ed41f9a0b4f85844a63abd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75091d4322ed41f9a0b4f85844a63abd" member_type: VOTER } }
I20260505 20:18:08.673723 29915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:08.673796 29914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:08.678097 29918 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:08.683233 29918 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:08.688741 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:08.696424 29918 catalog_manager.cc:1357] Generated new cluster ID: f7071a89aab04ff196d833a524acc6ee
I20260505 20:18:08.696805 29918 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:08.728425 29918 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:08.730314 29918 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:08.755990 29918 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd: Generated new TSK 0
I20260505 20:18:08.756912 29918 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:08.763132 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:08.770869 29931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:08.771405 29932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:08.781103 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:08.781105 29934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:08.782478 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:08.782788 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:08.782972 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012288782945 us; error 0 us; skew 500 ppm
I20260505 20:18:08.783572 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:08.786513 28669 webserver.cc:492] Webserver started at http://127.27.255.65:33079/ using document root <none> and password file <none>
I20260505 20:18:08.787084 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:08.787312 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:08.787739 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:08.789129 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "25193801bd8c42228a686e9384439726"
format_stamp: "Formatted at 2026-05-05 20:18:08 on dist-test-slave-dnnj"
I20260505 20:18:08.794595 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260505 20:18:08.799206 29939 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:08.800381 28669 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260505 20:18:08.800745 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "25193801bd8c42228a686e9384439726"
format_stamp: "Formatted at 2026-05-05 20:18:08 on dist-test-slave-dnnj"
I20260505 20:18:08.801040 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:08.831326 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:08.832687 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:08.834484 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:08.837163 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:08.837430 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:08.837764 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:08.837954 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:08.885444 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:39053
I20260505 20:18:08.885648 30001 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:39053 every 8 connection(s)
I20260505 20:18:08.902311 30002 heartbeater.cc:344] Connected to a master server at 127.27.255.126:43573
I20260505 20:18:08.902752 30002 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:08.903693 30002 heartbeater.cc:507] Master 127.27.255.126:43573 requested a full tablet report, sending...
I20260505 20:18:08.906028 29875 ts_manager.cc:194] Registered new tserver with Master: 25193801bd8c42228a686e9384439726 (127.27.255.65:39053)
I20260505 20:18:08.906689 28669 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.017407708s
I20260505 20:18:08.908015 29875 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33396
I20260505 20:18:08.932294 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:08.958643 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:08.977911 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:08.986498 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260505 20:18:08.989306 28669 tablet_server.cc:94] deduplicated master addresses: 127.27.255.126:43573
W20260505 20:18:08.992914 30012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:08.994822 30013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:08.994889 30015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:08.995080 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:08.996323 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:08.996510 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:08.996697 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012288996683 us; error 0 us; skew 500 ppm
I20260505 20:18:08.997279 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:08.999718 28669 webserver.cc:492] Webserver started at http://127.27.255.65:33079/ using document root <none> and password file <none>
I20260505 20:18:09.000253 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:09.000411 28669 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260505 20:18:09.004040 28669 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260505 20:18:09.006857 30020 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:09.007719 28669 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260505 20:18:09.008040 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "25193801bd8c42228a686e9384439726"
format_stamp: "Formatted at 2026-05-05 20:18:08 on dist-test-slave-dnnj"
I20260505 20:18:09.008437 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:09.034170 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:09.035195 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:09.036643 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:09.039279 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:09.039567 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:09.040130 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:09.040548 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:09.088315 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:39053
I20260505 20:18:09.088470 30082 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:39053 every 8 connection(s)
I20260505 20:18:09.094738 28669 tablet_server.cc:179] TabletServer@127.27.255.65:39053 shutting down...
I20260505 20:18:09.109855 30083 heartbeater.cc:344] Connected to a master server at 127.27.255.126:43573
I20260505 20:18:09.110494 30083 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:09.112838 30083 heartbeater.cc:507] Master 127.27.255.126:43573 requested a full tablet report, sending...
I20260505 20:18:09.115101 29875 ts_manager.cc:194] Re-registered known tserver with Master: 25193801bd8c42228a686e9384439726 (127.27.255.65:39053)
I20260505 20:18:09.117556 29875 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33406
I20260505 20:18:09.123734 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:09.139766 28669 tablet_server.cc:196] TabletServer@127.27.255.65:39053 shutdown complete.
I20260505 20:18:09.147388 28669 master.cc:562] Master@127.27.255.126:43573 shutting down...
I20260505 20:18:09.170256 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:09.170822 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:09.171304 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P 75091d4322ed41f9a0b4f85844a63abd: stopping tablet replica
I20260505 20:18:09.180991 28669 master.cc:584] Master@127.27.255.126:43573 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (673 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20260505 20:18:09.206017 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:40673,127.27.255.125:45869,127.27.255.124:41987
I20260505 20:18:09.207244 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:09.212759 30090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:09.214195 30091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:09.215975 30093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.216563 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:09.217633 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:09.217828 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:09.218111 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012289218084 us; error 0 us; skew 500 ppm
I20260505 20:18:09.218875 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:09.221622 28669 webserver.cc:492] Webserver started at http://127.27.255.126:33753/ using document root <none> and password file <none>
I20260505 20:18:09.222196 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:09.222393 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:09.222682 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:09.223776 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "2e02ce4bdc754528a079bdb143064843"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:09.228477 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260505 20:18:09.232151 30098 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:09.233301 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260505 20:18:09.233677 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "2e02ce4bdc754528a079bdb143064843"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:09.233996 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:09.260407 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:09.261693 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:09.303524 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:40673
I20260505 20:18:09.303638 30149 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:40673 every 8 connection(s)
I20260505 20:18:09.308109 30150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:09.308389 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:09.315356 30152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:09.316601 30153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.316651 30150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } has no permanent_uuid. Determining permanent_uuid...
W20260505 20:18:09.319674 30155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.322140 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:09.323594 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:09.323957 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:09.324170 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012289324146 us; error 0 us; skew 500 ppm
I20260505 20:18:09.325098 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:09.328860 28669 webserver.cc:492] Webserver started at http://127.27.255.125:41315/ using document root <none> and password file <none>
I20260505 20:18:09.329442 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:09.329723 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:09.330046 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:09.331789 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-1-root/instance:
uuid: "ef83e6b036a84a31b76b0df6a080dbce"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:09.334954 30150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } has no permanent_uuid. Determining permanent_uuid...
I20260505 20:18:09.337554 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
W20260505 20:18:09.337729 30100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.255.125:45869: connect: Connection refused (error 111)
W20260505 20:18:09.341826 30150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.255.125:45869: Network error: Client connection negotiation failed: client connection to 127.27.255.125:45869: connect: Connection refused (error 111)
I20260505 20:18:09.342012 30163 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:09.342964 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260505 20:18:09.343245 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-1-root
uuid: "ef83e6b036a84a31b76b0df6a080dbce"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:09.343513 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:09.374303 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:09.375306 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:09.380445 30150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } attempt: 1
W20260505 20:18:09.387640 30150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.255.125:45869: Network error: Client connection negotiation failed: client connection to 127.27.255.125:45869: connect: Connection refused (error 111)
I20260505 20:18:09.423406 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.125:45869
I20260505 20:18:09.423707 30214 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.125:45869 every 8 connection(s)
I20260505 20:18:09.428385 30150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } attempt: 2
I20260505 20:18:09.429391 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260505 20:18:09.430022 30215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20260505 20:18:09.437371 30218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.441262 30215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } has no permanent_uuid. Determining permanent_uuid...
I20260505 20:18:09.449002 30150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } has no permanent_uuid. Determining permanent_uuid...
W20260505 20:18:09.450166 30219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.450776 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:09.454893 30221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.458127 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:09.458470 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
W20260505 20:18:09.458572 30150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.255.124:41987: Network error: Client connection negotiation failed: client connection to 127.27.255.124:41987: connect: Connection refused (error 111)
I20260505 20:18:09.458827 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012289458697 us; error 0 us; skew 500 ppm
I20260505 20:18:09.460528 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:09.464015 30215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } has no permanent_uuid. Determining permanent_uuid...
I20260505 20:18:09.465890 28669 webserver.cc:492] Webserver started at http://127.27.255.124:35143/ using document root <none> and password file <none>
I20260505 20:18:09.466468 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:09.466763 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:09.467092 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:09.468793 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-2-root/instance:
uuid: "6f405746a59b4a56ad59181504704852"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:09.474191 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260505 20:18:09.479219 30227 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:09.480383 30215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } has no permanent_uuid. Determining permanent_uuid...
I20260505 20:18:09.480861 28669 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260505 20:18:09.481573 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-2-root
uuid: "6f405746a59b4a56ad59181504704852"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:09.481945 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260505 20:18:09.487648 30215 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.255.124:41987: Network error: Client connection negotiation failed: client connection to 127.27.255.124:41987: connect: Connection refused (error 111)
I20260505 20:18:09.494185 30150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } attempt: 1
W20260505 20:18:09.498983 30150 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.255.124:41987: Network error: Client connection negotiation failed: client connection to 127.27.255.124:41987: connect: Connection refused (error 111)
I20260505 20:18:09.512452 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:09.513665 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:09.521302 30215 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } attempt: 1
W20260505 20:18:09.527519 30215 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.255.124:41987: Network error: Client connection negotiation failed: client connection to 127.27.255.124:41987: connect: Connection refused (error 111)
I20260505 20:18:09.556118 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.124:41987
I20260505 20:18:09.556309 30279 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.124:41987 every 8 connection(s)
I20260505 20:18:09.559937 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:09.560853 30280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:09.566077 30215 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } attempt: 2
I20260505 20:18:09.567832 30280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } has no permanent_uuid. Determining permanent_uuid...
I20260505 20:18:09.574985 30150 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } attempt: 2
I20260505 20:18:09.583302 30280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } has no permanent_uuid. Determining permanent_uuid...
I20260505 20:18:09.592597 30215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce: Bootstrap starting.
I20260505 20:18:09.598743 30280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } has no permanent_uuid. Determining permanent_uuid...
I20260505 20:18:09.601118 30150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843: Bootstrap starting.
I20260505 20:18:09.601118 30215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:09.614212 30215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce: No bootstrap required, opened a new log
I20260505 20:18:09.614804 30150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:09.617143 30215 raft_consensus.cc:359] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:09.618109 30215 raft_consensus.cc:385] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:09.618431 30215 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef83e6b036a84a31b76b0df6a080dbce, State: Initialized, Role: FOLLOWER
I20260505 20:18:09.619127 30215 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:09.620239 30150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843: No bootstrap required, opened a new log
I20260505 20:18:09.622148 30280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: Bootstrap starting.
I20260505 20:18:09.622926 30215 sys_catalog.cc:565] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:09.623672 30150 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:09.624782 30150 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:09.625233 30150 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e02ce4bdc754528a079bdb143064843, State: Initialized, Role: FOLLOWER
I20260505 20:18:09.626430 30150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:09.627681 30287 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:09.629594 30287 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:09.629889 30280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:09.630015 30287 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:09.630453 30287 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:09.635259 30290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:09.636384 30290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:09.636518 30150 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:09.636906 30125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef83e6b036a84a31b76b0df6a080dbce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843" is_pre_election: true
I20260505 20:18:09.637647 30125 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef83e6b036a84a31b76b0df6a080dbce in term 0.
I20260505 20:18:09.638944 30165 leader_election.cc:304] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843, ef83e6b036a84a31b76b0df6a080dbce; no voters:
I20260505 20:18:09.639685 30288 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260505 20:18:09.640021 30280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: No bootstrap required, opened a new log
I20260505 20:18:09.640159 30288 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:09.640667 30288 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:09.641215 30287 leader_election.cc:290] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:09.643780 30280 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:09.644744 30280 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:09.645531 30280 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f405746a59b4a56ad59181504704852, State: Initialized, Role: FOLLOWER
I20260505 20:18:09.647913 30280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:09.653509 30288 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:09.655514 30255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef83e6b036a84a31b76b0df6a080dbce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f405746a59b4a56ad59181504704852" is_pre_election: true
I20260505 20:18:09.656237 30255 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef83e6b036a84a31b76b0df6a080dbce in term 0.
I20260505 20:18:09.657475 30280 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:09.659232 30125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef83e6b036a84a31b76b0df6a080dbce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843"
I20260505 20:18:09.659422 30300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:09.659955 30125 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:09.660162 30300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:09.671669 30255 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef83e6b036a84a31b76b0df6a080dbce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f405746a59b4a56ad59181504704852"
I20260505 20:18:09.672457 30255 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:09.682708 30288 leader_election.cc:290] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 1 election: Requested vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:09.684827 30255 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef83e6b036a84a31b76b0df6a080dbce in term 1.
I20260505 20:18:09.686951 30165 leader_election.cc:304] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f405746a59b4a56ad59181504704852, ef83e6b036a84a31b76b0df6a080dbce; no voters:
I20260505 20:18:09.688064 30288 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:09.689020 30288 raft_consensus.cc:697] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 1 LEADER]: Becoming Leader. State: Replica: ef83e6b036a84a31b76b0df6a080dbce, State: Running, Role: LEADER
I20260505 20:18:09.690246 30288 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:09.669082 30125 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef83e6b036a84a31b76b0df6a080dbce in term 1.
I20260505 20:18:09.700505 30287 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef83e6b036a84a31b76b0df6a080dbce. Latest consensus state: current_term: 1 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:09.701298 30287 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:09.708664 30312 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:09.717890 30312 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:09.726212 30318 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260505 20:18:09.735862 30255 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 1 FOLLOWER]: Refusing update from remote peer ef83e6b036a84a31b76b0df6a080dbce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260505 20:18:09.738219 30288 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260505 20:18:09.745792 30125 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 1 FOLLOWER]: Refusing update from remote peer ef83e6b036a84a31b76b0df6a080dbce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260505 20:18:09.746796 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20260505 20:18:09.747057 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20260505 20:18:09.748103 30287 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260505 20:18:09.749606 30326 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260505 20:18:09.750007 30326 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260505 20:18:09.763746 30300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef83e6b036a84a31b76b0df6a080dbce. Latest consensus state: current_term: 1 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:09.765316 30300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:09.772647 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260505 20:18:09.775229 30287 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
E20260505 20:18:09.776764 28669 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260505 20:18:09.777065 30287 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:09.783386 30312 catalog_manager.cc:1357] Generated new cluster ID: 1cd94111d53d4222a299fb808db64b51
I20260505 20:18:09.783823 30312 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:09.783365 30290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef83e6b036a84a31b76b0df6a080dbce. Latest consensus state: current_term: 1 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:09.784278 30290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:09.787209 30287 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:09.788267 30287 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:09.788508 30300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:09.789211 30300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:09.795037 30330 catalog_manager.cc:1269] Loaded cluster ID: 1cd94111d53d4222a299fb808db64b51
I20260505 20:18:09.794719 30329 mvcc.cc:204] Tried to move back new op lower bound from 7282738338739785728 to 7282738338594377728. Current Snapshot: MvccSnapshot[applied={T|T < 7282738338739785728}]
I20260505 20:18:09.795609 30330 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: loading cluster ID for follower catalog manager: success
W20260505 20:18:09.799590 30330 catalog_manager.cc:1586] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260505 20:18:09.800354 30330 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260505 20:18:09.801391 30290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:09.802141 30290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
W20260505 20:18:09.803153 30335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:09.807930 30336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.810873 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:09.811697 30339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.812785 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:09.813064 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:09.813256 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012289813240 us; error 0 us; skew 500 ppm
I20260505 20:18:09.813944 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:09.816900 28669 webserver.cc:492] Webserver started at http://127.27.255.65:42513/ using document root <none> and password file <none>
I20260505 20:18:09.817461 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:09.817725 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:09.818022 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:09.819372 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "e16e42524090435eb7561f44a8815878"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:09.827450 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260505 20:18:09.827601 30312 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:09.829403 30312 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:09.832372 30344 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:09.833360 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:09.833732 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "e16e42524090435eb7561f44a8815878"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:09.834067 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:09.854784 30312 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce: Generated new TSK 0
I20260505 20:18:09.855461 30312 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:09.875598 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:09.876889 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:09.878458 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:09.880821 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:09.881091 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:09.881367 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:09.881590 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260505 20:18:09.926064 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:36045
I20260505 20:18:09.926143 30407 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:36045 every 8 connection(s)
I20260505 20:18:09.934410 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260505 20:18:09.937038 28669 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260505 20:18:09.952306 30415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.952729 30409 heartbeater.cc:344] Connected to a master server at 127.27.255.125:45869
I20260505 20:18:09.953426 30409 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:09.954399 30409 heartbeater.cc:507] Master 127.27.255.125:45869 requested a full tablet report, sending...
I20260505 20:18:09.954732 30408 heartbeater.cc:344] Connected to a master server at 127.27.255.124:41987
I20260505 20:18:09.955080 30408 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:09.956058 30408 heartbeater.cc:507] Master 127.27.255.124:41987 requested a full tablet report, sending...
W20260505 20:18:09.958233 30416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.958976 30180 ts_manager.cc:194] Registered new tserver with Master: e16e42524090435eb7561f44a8815878 (127.27.255.65:36045)
I20260505 20:18:09.961516 30245 ts_manager.cc:194] Registered new tserver with Master: e16e42524090435eb7561f44a8815878 (127.27.255.65:36045)
I20260505 20:18:09.961687 30180 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49958
I20260505 20:18:09.961772 30410 heartbeater.cc:344] Connected to a master server at 127.27.255.126:40673
I20260505 20:18:09.962255 30410 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:09.963265 30410 heartbeater.cc:507] Master 127.27.255.126:40673 requested a full tablet report, sending...
W20260505 20:18:09.965096 30418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:09.966976 30115 ts_manager.cc:194] Registered new tserver with Master: e16e42524090435eb7561f44a8815878 (127.27.255.65:36045)
I20260505 20:18:09.968245 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:09.969079 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:09.969303 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:09.969563 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012289969542 us; error 0 us; skew 500 ppm
I20260505 20:18:09.970273 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:09.972877 28669 webserver.cc:492] Webserver started at http://127.27.255.66:42299/ using document root <none> and password file <none>
I20260505 20:18:09.973376 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:09.973642 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:09.973920 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:09.975322 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-1-root/instance:
uuid: "ecce4a805d624a32b52d29f1cb860010"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:09.980978 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260505 20:18:09.985003 30423 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:09.985949 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260505 20:18:09.986341 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-1-root
uuid: "ecce4a805d624a32b52d29f1cb860010"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:09.986654 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:10.010922 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:10.012240 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:10.013633 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:10.016995 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:10.017302 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:10.017765 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:10.018467 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260505 20:18:10.066458 30409 heartbeater.cc:499] Master 127.27.255.125:45869 was elected leader, sending a full tablet report...
I20260505 20:18:10.113675 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.66:42685
I20260505 20:18:10.113958 30485 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.66:42685 every 8 connection(s)
I20260505 20:18:10.123137 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260505 20:18:10.126554 28669 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260505 20:18:10.138711 30486 heartbeater.cc:344] Connected to a master server at 127.27.255.124:41987
I20260505 20:18:10.139214 30486 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:10.140105 30486 heartbeater.cc:507] Master 127.27.255.124:41987 requested a full tablet report, sending...
I20260505 20:18:10.142992 30245 ts_manager.cc:194] Registered new tserver with Master: ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:10.145844 30487 heartbeater.cc:344] Connected to a master server at 127.27.255.125:45869
I20260505 20:18:10.146854 30487 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:10.148049 30487 heartbeater.cc:507] Master 127.27.255.125:45869 requested a full tablet report, sending...
W20260505 20:18:10.150816 30494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:10.150851 30180 ts_manager.cc:194] Registered new tserver with Master: ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:10.152182 30488 heartbeater.cc:344] Connected to a master server at 127.27.255.126:40673
I20260505 20:18:10.152608 30488 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:10.152954 30180 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49964
I20260505 20:18:10.153537 30488 heartbeater.cc:507] Master 127.27.255.126:40673 requested a full tablet report, sending...
W20260505 20:18:10.156720 30493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:10.158208 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:10.158788 30115 ts_manager.cc:194] Registered new tserver with Master: ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
W20260505 20:18:10.159240 30496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:10.160614 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:10.160934 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:10.161187 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012290161125 us; error 0 us; skew 500 ppm
I20260505 20:18:10.162042 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:10.165968 28669 webserver.cc:492] Webserver started at http://127.27.255.67:43339/ using document root <none> and password file <none>
I20260505 20:18:10.166622 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:10.166868 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:10.167380 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:10.168846 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-2-root/instance:
uuid: "7c3588319984403a88bdce106eea1e56"
format_stamp: "Formatted at 2026-05-05 20:18:10 on dist-test-slave-dnnj"
I20260505 20:18:10.175518 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260505 20:18:10.180238 30501 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:10.181192 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:10.181466 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-2-root
uuid: "7c3588319984403a88bdce106eea1e56"
format_stamp: "Formatted at 2026-05-05 20:18:10 on dist-test-slave-dnnj"
I20260505 20:18:10.181798 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:10.200783 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:10.201984 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:10.204298 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:10.207295 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:10.207523 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:10.207953 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:10.208186 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:10.257084 30487 heartbeater.cc:499] Master 127.27.255.125:45869 was elected leader, sending a full tablet report...
I20260505 20:18:10.304965 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.67:40263
I20260505 20:18:10.305058 30563 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.67:40263 every 8 connection(s)
I20260505 20:18:10.327761 30564 heartbeater.cc:344] Connected to a master server at 127.27.255.124:41987
I20260505 20:18:10.328183 30564 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:10.329151 30564 heartbeater.cc:507] Master 127.27.255.124:41987 requested a full tablet report, sending...
I20260505 20:18:10.329828 30565 heartbeater.cc:344] Connected to a master server at 127.27.255.125:45869
I20260505 20:18:10.330262 30565 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:10.331043 30565 heartbeater.cc:507] Master 127.27.255.125:45869 requested a full tablet report, sending...
I20260505 20:18:10.332769 30245 ts_manager.cc:194] Registered new tserver with Master: 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:10.333197 30180 ts_manager.cc:194] Registered new tserver with Master: 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:10.334226 30566 heartbeater.cc:344] Connected to a master server at 127.27.255.126:40673
I20260505 20:18:10.334673 30566 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:10.335115 30180 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49976
I20260505 20:18:10.335480 30566 heartbeater.cc:507] Master 127.27.255.126:40673 requested a full tablet report, sending...
I20260505 20:18:10.339401 30115 ts_manager.cc:194] Registered new tserver with Master: 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:10.339499 28669 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.025653154s
I20260505 20:18:10.386747 30179 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20260505 20:18:10.388640 30179 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260505 20:18:10.437752 30565 heartbeater.cc:499] Master 127.27.255.125:45869 was elected leader, sending a full tablet report...
I20260505 20:18:10.493875 30451 tablet_service.cc:1511] Processing CreateTablet for tablet 10ea1d60fc224e79a768017a09747cc0 (DEFAULT_TABLE table=test-table [id=8cddca7fffa74a7983714816b4d4457d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:10.494267 30529 tablet_service.cc:1511] Processing CreateTablet for tablet 10ea1d60fc224e79a768017a09747cc0 (DEFAULT_TABLE table=test-table [id=8cddca7fffa74a7983714816b4d4457d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:10.494660 30528 tablet_service.cc:1511] Processing CreateTablet for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc (DEFAULT_TABLE table=test-table [id=8cddca7fffa74a7983714816b4d4457d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:10.495658 30451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10ea1d60fc224e79a768017a09747cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:10.495743 30529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10ea1d60fc224e79a768017a09747cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:10.497816 30450 tablet_service.cc:1511] Processing CreateTablet for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc (DEFAULT_TABLE table=test-table [id=8cddca7fffa74a7983714816b4d4457d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:10.502743 30449 tablet_service.cc:1511] Processing CreateTablet for tablet 6f497a2e79934ecc9e9ab47b7d98a9d1 (DEFAULT_TABLE table=test-table [id=8cddca7fffa74a7983714816b4d4457d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:10.507151 30528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:10.507624 30450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:10.509694 30449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f497a2e79934ecc9e9ab47b7d98a9d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:10.512888 30527 tablet_service.cc:1511] Processing CreateTablet for tablet 6f497a2e79934ecc9e9ab47b7d98a9d1 (DEFAULT_TABLE table=test-table [id=8cddca7fffa74a7983714816b4d4457d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:10.514604 30527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f497a2e79934ecc9e9ab47b7d98a9d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:10.520066 30373 tablet_service.cc:1511] Processing CreateTablet for tablet 10ea1d60fc224e79a768017a09747cc0 (DEFAULT_TABLE table=test-table [id=8cddca7fffa74a7983714816b4d4457d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:10.520984 30371 tablet_service.cc:1511] Processing CreateTablet for tablet 6f497a2e79934ecc9e9ab47b7d98a9d1 (DEFAULT_TABLE table=test-table [id=8cddca7fffa74a7983714816b4d4457d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:10.522043 30373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10ea1d60fc224e79a768017a09747cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:10.523430 30372 tablet_service.cc:1511] Processing CreateTablet for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc (DEFAULT_TABLE table=test-table [id=8cddca7fffa74a7983714816b4d4457d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:10.529886 30371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f497a2e79934ecc9e9ab47b7d98a9d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:10.532313 30372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:10.573345 30594 tablet_bootstrap.cc:492] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010: Bootstrap starting.
I20260505 20:18:10.580376 30593 tablet_bootstrap.cc:492] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56: Bootstrap starting.
I20260505 20:18:10.620966 30593 tablet_bootstrap.cc:654] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:10.622915 30595 tablet_bootstrap.cc:492] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878: Bootstrap starting.
I20260505 20:18:10.626670 30594 tablet_bootstrap.cc:654] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:10.628530 30593 tablet_bootstrap.cc:492] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56: No bootstrap required, opened a new log
I20260505 20:18:10.629081 30593 ts_tablet_manager.cc:1403] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.016s
I20260505 20:18:10.631323 30595 tablet_bootstrap.cc:654] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:10.632640 30593 raft_consensus.cc:359] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.633486 30593 raft_consensus.cc:385] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:10.633823 30593 raft_consensus.cc:740] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3588319984403a88bdce106eea1e56, State: Initialized, Role: FOLLOWER
I20260505 20:18:10.634964 30594 tablet_bootstrap.cc:492] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010: No bootstrap required, opened a new log
I20260505 20:18:10.634816 30593 consensus_queue.cc:260] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.635653 30594 ts_tablet_manager.cc:1403] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.002s
I20260505 20:18:10.640385 30593 ts_tablet_manager.cc:1434] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260505 20:18:10.640280 30594 raft_consensus.cc:359] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.641278 30594 raft_consensus.cc:385] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:10.641626 30594 raft_consensus.cc:740] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce4a805d624a32b52d29f1cb860010, State: Initialized, Role: FOLLOWER
I20260505 20:18:10.642403 30594 consensus_queue.cc:260] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.650897 30593 tablet_bootstrap.cc:492] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56: Bootstrap starting.
I20260505 20:18:10.654054 30594 ts_tablet_manager.cc:1434] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010: Time spent starting tablet: real 0.018s user 0.005s sys 0.009s
I20260505 20:18:10.657543 30594 tablet_bootstrap.cc:492] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010: Bootstrap starting.
I20260505 20:18:10.660028 30593 tablet_bootstrap.cc:654] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:10.662426 30595 tablet_bootstrap.cc:492] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878: No bootstrap required, opened a new log
I20260505 20:18:10.662971 30595 ts_tablet_manager.cc:1403] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878: Time spent bootstrapping tablet: real 0.045s user 0.020s sys 0.009s
I20260505 20:18:10.668922 30594 tablet_bootstrap.cc:654] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:10.668823 30595 raft_consensus.cc:359] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.669775 30595 raft_consensus.cc:385] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:10.670104 30595 raft_consensus.cc:740] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16e42524090435eb7561f44a8815878, State: Initialized, Role: FOLLOWER
I20260505 20:18:10.670817 30595 consensus_queue.cc:260] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.675815 30600 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:10.676532 30600 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.680126 30595 ts_tablet_manager.cc:1434] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
I20260505 20:18:10.682726 30600 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:10.683362 30594 tablet_bootstrap.cc:492] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010: No bootstrap required, opened a new log
I20260505 20:18:10.685336 30594 ts_tablet_manager.cc:1403] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.010s
I20260505 20:18:10.688750 30595 tablet_bootstrap.cc:492] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878: Bootstrap starting.
I20260505 20:18:10.688400 30594 raft_consensus.cc:359] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.689167 30594 raft_consensus.cc:385] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:10.689463 30594 raft_consensus.cc:740] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce4a805d624a32b52d29f1cb860010, State: Initialized, Role: FOLLOWER
I20260505 20:18:10.690156 30594 consensus_queue.cc:260] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.695570 30594 ts_tablet_manager.cc:1434] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260505 20:18:10.695971 30595 tablet_bootstrap.cc:654] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:10.697010 30593 tablet_bootstrap.cc:492] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56: No bootstrap required, opened a new log
I20260505 20:18:10.697829 30593 ts_tablet_manager.cc:1403] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.015s
I20260505 20:18:10.697975 30602 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:10.697083 30594 tablet_bootstrap.cc:492] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010: Bootstrap starting.
I20260505 20:18:10.698556 30602 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.702729 30593 raft_consensus.cc:359] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.711975 30593 raft_consensus.cc:385] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:10.712409 30593 raft_consensus.cc:740] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3588319984403a88bdce106eea1e56, State: Initialized, Role: FOLLOWER
I20260505 20:18:10.703272 30602 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:10.724619 30593 consensus_queue.cc:260] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.727828 30593 ts_tablet_manager.cc:1434] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56: Time spent starting tablet: real 0.030s user 0.006s sys 0.002s
I20260505 20:18:10.729238 30593 tablet_bootstrap.cc:492] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56: Bootstrap starting.
I20260505 20:18:10.744064 30594 tablet_bootstrap.cc:654] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:10.755700 30326 catalog_manager.cc:1269] Loaded cluster ID: 1cd94111d53d4222a299fb808db64b51
I20260505 20:18:10.755928 30593 tablet_bootstrap.cc:654] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:10.756073 30326 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843: loading cluster ID for follower catalog manager: success
I20260505 20:18:10.761927 30599 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:10.762518 30599 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.764101 30595 tablet_bootstrap.cc:492] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878: No bootstrap required, opened a new log
I20260505 20:18:10.764755 30595 ts_tablet_manager.cc:1403] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878: Time spent bootstrapping tablet: real 0.076s user 0.015s sys 0.004s
I20260505 20:18:10.764520 30461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:10.765470 30461 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 0.
I20260505 20:18:10.766837 30599 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:10.767341 30345 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:10.768627 30606 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260505 20:18:10.768603 30595 raft_consensus.cc:359] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.769470 30606 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:10.769959 30595 raft_consensus.cc:385] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:10.771149 30599 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:10.771624 30599 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.772126 30595 raft_consensus.cc:740] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16e42524090435eb7561f44a8815878, State: Initialized, Role: FOLLOWER
I20260505 20:18:10.773819 30606 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:10.774474 30599 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:10.776454 30600 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:10.778213 30600 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.781131 30600 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:10.783797 30595 consensus_queue.cc:260] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.793895 30383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:10.802269 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:10.803287 30539 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 0.
I20260505 20:18:10.797287 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:10.820060 30330 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: acquiring CA information for follower catalog manager: success
I20260505 20:18:10.825560 30326 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843: acquiring CA information for follower catalog manager: success
I20260505 20:18:10.835129 30606 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:10.956602 30383 raft_consensus.cc:2393] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecce4a805d624a32b52d29f1cb860010 in current term 1: Already voted for candidate e16e42524090435eb7561f44a8815878 in this term.
I20260505 20:18:10.954864 30602 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:10.976944 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:10.972409 30602 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.003026 30606 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 1 election: Requested vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:11.003815 30330 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260505 20:18:11.003599 30424 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.255.65:36045, user_credentials={real_user=slave}} blocked reactor thread for 81925.5us
I20260505 20:18:11.008611 30461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:11.009369 30461 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Advancing to term 1
W20260505 20:18:11.010032 30383 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55612 (request call id 0) took 221 ms (client timeout 210 ms). Trace:
W20260505 20:18:11.010470 30383 rpcz_store.cc:269] 0505 20:18:10.788908 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:10.790924 (+ 2016us) service_pool.cc:224] Handling call
0505 20:18:11.009795 (+218871us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400320354. stack: 00007fe78bbcdc83 00007fe78bbcdb80 00007fe78b51b574 00000000003bff74 00000000003bff09 00007fe79256d951 00007fe7926086da 00007fe79461d1d7 00007fe7920b1834 00007fe7920b17c2 00007fe7920b16f1 00007fe7920b166c 00007fe7920b0922 00007fe7907d4e4c 00007fe7907d4286 00007fe7907d3c2f
0505 20:18:11.009963 (+ 168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264616704}
I20260505 20:18:11.002182 30382 raft_consensus.cc:2468] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 0.
I20260505 20:18:11.017187 30381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:11.020022 30381 raft_consensus.cc:2393] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecce4a805d624a32b52d29f1cb860010 in current term 1: Already voted for candidate e16e42524090435eb7561f44a8815878 in this term.
W20260505 20:18:11.020046 30382 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55612 (request call id 1) took 230 ms (client timeout 284 ms). Trace:
W20260505 20:18:11.023336 30382 rpcz_store.cc:269] 0505 20:18:10.789311 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:10.790786 (+ 1475us) service_pool.cc:224] Handling call
0505 20:18:11.019886 (+229100us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400340154. stack: 00007fe78bbcdc83 00007fe78bbcdb80 00007fe78b51b574 00000000003bff74 00000000003bff09 00007fe79256d951 00007fe7926086da 00007fe79461d1d7 00007fe7920b1834 00007fe7920b17c2 00007fe7920b16f1 00007fe7920b166c 00007fe7920b0922 00007fe7907d4e4c 00007fe7907d4286 00007fe7907d3c2f
0505 20:18:11.019990 (+ 104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":436280576}
I20260505 20:18:11.025848 30424 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:11.027860 30600 raft_consensus.cc:2804] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260505 20:18:11.028486 30593 tablet_bootstrap.cc:492] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56: No bootstrap required, opened a new log
I20260505 20:18:11.028755 30600 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:11.029132 30593 ts_tablet_manager.cc:1403] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56: Time spent bootstrapping tablet: real 0.300s user 0.032s sys 0.042s
I20260505 20:18:11.029538 30600 raft_consensus.cc:3060] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:11.029978 30595 ts_tablet_manager.cc:1434] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878: Time spent starting tablet: real 0.265s user 0.030s sys 0.026s
I20260505 20:18:11.031226 30595 tablet_bootstrap.cc:492] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878: Bootstrap starting.
I20260505 20:18:11.032688 30593 raft_consensus.cc:359] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.039933 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:11.040758 30539 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 0.
I20260505 20:18:11.041725 30614 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:11.041997 30602 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:11.042515 30614 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:11.043701 30595 tablet_bootstrap.cc:654] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:11.033953 30593 raft_consensus.cc:385] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:11.045673 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
W20260505 20:18:11.046254 30425 leader_election.cc:343] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263): Illegal state: must be running to vote when last-logged opid is not known
I20260505 20:18:11.061517 30538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
W20260505 20:18:11.084416 30425 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.255.67:40263, user_credentials={real_user=slave}} blocked reactor thread for 45813.3us
I20260505 20:18:11.084728 30461 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 1.
I20260505 20:18:11.086274 30345 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:11.086692 30425 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, ecce4a805d624a32b52d29f1cb860010; no voters: e16e42524090435eb7561f44a8815878
I20260505 20:18:11.087540 30602 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:11.088194 30602 raft_consensus.cc:697] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 1 LEADER]: Becoming Leader. State: Replica: e16e42524090435eb7561f44a8815878, State: Running, Role: LEADER
I20260505 20:18:11.089150 30602 consensus_queue.cc:237] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.090859 30614 leader_election.cc:290] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869)
I20260505 20:18:11.085597 30326 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260505 20:18:11.082775 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56"
I20260505 20:18:11.096798 30461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:11.057957 30593 raft_consensus.cc:740] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3588319984403a88bdce106eea1e56, State: Initialized, Role: FOLLOWER
I20260505 20:18:11.098065 30539 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:11.095685 30620 raft_consensus.cc:2764] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260505 20:18:11.092487 30594 tablet_bootstrap.cc:492] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010: No bootstrap required, opened a new log
I20260505 20:18:11.099713 30459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:11.101420 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:11.102496 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce" is_pre_election: true
W20260505 20:18:11.106369 30502 leader_election.cc:343] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685): Illegal state: must be running to vote when last-logged opid is not known
I20260505 20:18:11.108246 30124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843" is_pre_election: true
I20260505 20:18:11.111117 30124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f405746a59b4a56ad59181504704852 in term 1.
I20260505 20:18:11.112454 30459 raft_consensus.cc:2393] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c3588319984403a88bdce106eea1e56 in current term 1: Already voted for candidate e16e42524090435eb7561f44a8815878 in this term.
I20260505 20:18:11.109006 30594 ts_tablet_manager.cc:1403] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010: Time spent bootstrapping tablet: real 0.412s user 0.022s sys 0.030s
I20260505 20:18:11.113593 30230 leader_election.cc:304] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843, 6f405746a59b4a56ad59181504704852; no voters:
I20260505 20:18:11.116259 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:11.117203 30614 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260505 20:18:11.118839 30614 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:11.119336 30614 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:11.102802 30593 consensus_queue.cc:260] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
W20260505 20:18:11.124177 30502 leader_election.cc:343] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e16e42524090435eb7561f44a8815878 (127.27.255.65:36045): Illegal state: must be running to vote when last-logged opid is not known
I20260505 20:18:11.124905 30537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:11.125115 30502 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:11.125262 30538 raft_consensus.cc:2468] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 0.
I20260505 20:18:11.127861 30539 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 1.
I20260505 20:18:11.132709 30594 raft_consensus.cc:359] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.128811 30619 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:11.134492 30599 raft_consensus.cc:2749] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260505 20:18:11.134207 30619 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:11.135044 30593 ts_tablet_manager.cc:1434] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56: Time spent starting tablet: real 0.105s user 0.009s sys 0.000s
I20260505 20:18:11.136689 30594 raft_consensus.cc:385] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:11.137027 30594 raft_consensus.cc:740] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecce4a805d624a32b52d29f1cb860010, State: Initialized, Role: FOLLOWER
I20260505 20:18:11.137441 30595 tablet_bootstrap.cc:492] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878: No bootstrap required, opened a new log
I20260505 20:18:11.137756 30594 consensus_queue.cc:260] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.137988 30595 ts_tablet_manager.cc:1403] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878: Time spent bootstrapping tablet: real 0.107s user 0.023s sys 0.008s
I20260505 20:18:11.140420 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce" is_pre_election: true
I20260505 20:18:11.148356 30594 ts_tablet_manager.cc:1434] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010: Time spent starting tablet: real 0.031s user 0.004s sys 0.009s
I20260505 20:18:11.144961 30600 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.151105 30460 raft_consensus.cc:2393] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e16e42524090435eb7561f44a8815878 in current term 1: Already voted for candidate ecce4a805d624a32b52d29f1cb860010 in this term.
I20260505 20:18:11.153537 30600 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 election: Requested vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:11.154682 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878"
I20260505 20:18:11.155342 30382 raft_consensus.cc:3060] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:11.155323 30595 raft_consensus.cc:359] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.156265 30595 raft_consensus.cc:385] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:11.156703 30595 raft_consensus.cc:740] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16e42524090435eb7561f44a8815878, State: Initialized, Role: FOLLOWER
I20260505 20:18:11.157385 30595 consensus_queue.cc:260] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.156306 30346 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878; no voters: ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:11.144387 30614 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:11.162667 30255 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 2 FOLLOWER]: Rejecting Update request from peer ef83e6b036a84a31b76b0df6a080dbce for earlier term 1. Current term is 2. Ops: []
I20260505 20:18:11.169878 30614 leader_election.cc:290] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 2 election: Requested vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869)
I20260505 20:18:11.164330 30619 leader_election.cc:290] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:11.168618 30124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843"
I20260505 20:18:11.159629 30595 ts_tablet_manager.cc:1434] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s
I20260505 20:18:11.166280 30254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "6f405746a59b4a56ad59181504704852" is_pre_election: true
I20260505 20:18:11.172842 30615 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 }, Status: INVALID_TERM, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260505 20:18:11.173187 30600 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:11.173357 30100 leader_election.cc:304] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843; no voters: 6f405746a59b4a56ad59181504704852, ef83e6b036a84a31b76b0df6a080dbce
I20260505 20:18:11.173997 30583 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 1 LEADER]: Stepping down as leader of term 1
I20260505 20:18:11.174266 30619 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260505 20:18:11.174427 30583 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ef83e6b036a84a31b76b0df6a080dbce, State: Running, Role: LEADER
I20260505 20:18:11.175222 30583 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:11.176225 30583 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:11.178313 30382 raft_consensus.cc:2468] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 1.
I20260505 20:18:11.179718 30424 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:11.171123 30179 catalog_manager.cc:5671] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 reported cstate change: term changed from 0 to 1, leader changed from <none> to e16e42524090435eb7561f44a8815878 (127.27.255.65). New cstate: current_term: 1 leader_uuid: "e16e42524090435eb7561f44a8815878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20260505 20:18:11.180462 30606 raft_consensus.cc:2764] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260505 20:18:11.180853 30620 raft_consensus.cc:2804] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:11.171375 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce"
I20260505 20:18:11.183338 30620 raft_consensus.cc:697] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 1 LEADER]: Becoming Leader. State: Replica: ecce4a805d624a32b52d29f1cb860010, State: Running, Role: LEADER
I20260505 20:18:11.184458 30620 consensus_queue.cc:237] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
E20260505 20:18:11.185381 30179 catalog_manager.cc:5826] Error updating tablets from e16e42524090435eb7561f44a8815878: Illegal state: replica ef83e6b036a84a31b76b0df6a080dbce is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e16e42524090435eb7561f44a8815878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } sequence_number: 10
W20260505 20:18:11.188514 30502 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.255.65:36045, user_credentials={real_user=slave}} blocked reactor thread for 64367.1us
I20260505 20:18:11.189642 30606 consensus_queue.cc:1048] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260505 20:18:11.189785 30502 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:11.189837 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56"
I20260505 20:18:11.190788 30539 raft_consensus.cc:3060] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:11.173744 30600 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.203047 30539 raft_consensus.cc:2468] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 1.
I20260505 20:18:11.202570 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:11.204563 30382 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 0.
I20260505 20:18:11.205967 30424 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:11.206790 30622 raft_consensus.cc:2749] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260505 20:18:11.209374 30600 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:11.209816 30625 raft_consensus.cc:2804] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260505 20:18:11.210350 30625 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:11.210707 30625 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:11.211521 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:11.220399 30606 consensus_queue.cc:1048] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260505 20:18:11.222399 30539 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 0.
I20260505 20:18:11.213258 30190 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f405746a59b4a56ad59181504704852 in term 2.
I20260505 20:18:11.239101 30230 leader_election.cc:304] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f405746a59b4a56ad59181504704852, ef83e6b036a84a31b76b0df6a080dbce; no voters: 2e02ce4bdc754528a079bdb143064843
W20260505 20:18:11.240290 30573 master_proxy_rpc.cc:203] Re-attempting IsCreateTableDone request to leader Master (127.27.255.125:45869)
I20260505 20:18:11.231007 30625 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.241838 30614 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 2 FOLLOWER]: Leader election won for term 2
I20260505 20:18:11.256150 30625 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 election: Requested vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:11.257061 30614 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 2 LEADER]: Becoming Leader. State: Replica: 6f405746a59b4a56ad59181504704852, State: Running, Role: LEADER
I20260505 20:18:11.257795 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878"
I20260505 20:18:11.258352 30538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56"
I20260505 20:18:11.259034 30538 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:11.258448 30382 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:11.271754 30382 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 1.
I20260505 20:18:11.272800 30614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:11.273195 30424 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:11.275326 30625 raft_consensus.cc:2804] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:11.276772 30538 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 1.
I20260505 20:18:11.278216 30625 raft_consensus.cc:697] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 1 LEADER]: Becoming Leader. State: Replica: ecce4a805d624a32b52d29f1cb860010, State: Running, Role: LEADER
I20260505 20:18:11.279016 30625 consensus_queue.cc:237] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:11.297699 30600 consensus_queue.cc:1048] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260505 20:18:11.311087 30600 consensus_queue.cc:1048] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260505 20:18:11.315315 30628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f405746a59b4a56ad59181504704852. Latest consensus state: current_term: 2 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:11.316125 30628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:11.341413 30634 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:11.372710 30634 catalog_manager.cc:679] Loaded metadata for table test-table [id=8cddca7fffa74a7983714816b4d4457d]
I20260505 20:18:11.372885 30600 consensus_queue.cc:1048] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260505 20:18:11.379210 30600 consensus_queue.cc:1048] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260505 20:18:11.381620 30634 tablet_loader.cc:96] loaded metadata for tablet 10ea1d60fc224e79a768017a09747cc0 (table test-table [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:11.384332 30634 tablet_loader.cc:96] loaded metadata for tablet 6f497a2e79934ecc9e9ab47b7d98a9d1 (table test-table [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:11.386765 30634 tablet_loader.cc:96] loaded metadata for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc (table test-table [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:11.388654 30634 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:11.394130 30124 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:11.407645 30190 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 2 FOLLOWER]: Refusing update from remote peer 6f405746a59b4a56ad59181504704852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260505 20:18:11.409246 30628 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260505 20:18:11.412082 30634 catalog_manager.cc:1269] Loaded cluster ID: 1cd94111d53d4222a299fb808db64b51
I20260505 20:18:11.412434 30634 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:11.414193 30583 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f405746a59b4a56ad59181504704852. Latest consensus state: current_term: 2 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:11.414929 30583 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:11.417248 30634 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:11.407681 30124 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 2 FOLLOWER]: Refusing update from remote peer 6f405746a59b4a56ad59181504704852: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260505 20:18:11.422600 30634 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: Loaded TSK: 0
I20260505 20:18:11.422932 30619 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:11.423466 30619 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:11.424750 30634 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:11.426489 30583 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:11.427259 30583 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:11.432765 30254 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "6f405746a59b4a56ad59181504704852" is_pre_election: true
I20260505 20:18:11.434051 30614 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260505 20:18:11.434998 30619 leader_election.cc:290] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:11.438697 30244 catalog_manager.cc:5671] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecce4a805d624a32b52d29f1cb860010 (127.27.255.66). New cstate: current_term: 1 leader_uuid: "ecce4a805d624a32b52d29f1cb860010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } } }
I20260505 20:18:11.440747 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 3 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce" is_pre_election: true
I20260505 20:18:11.443241 30100 leader_election.cc:304] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843; no voters: 6f405746a59b4a56ad59181504704852, ef83e6b036a84a31b76b0df6a080dbce
I20260505 20:18:11.444643 30619 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260505 20:18:11.446403 30628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:11.449664 30628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:11.477413 30645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:11.478312 30645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:11.478636 30408 heartbeater.cc:499] Master 127.27.255.124:41987 was elected leader, sending a full tablet report...
I20260505 20:18:11.478308 30619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f405746a59b4a56ad59181504704852. Latest consensus state: current_term: 2 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:11.479354 30619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:11.488469 30318 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260505 20:18:11.489725 30641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:11.490657 30641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:11.491228 30244 catalog_manager.cc:5671] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 reported cstate change: term changed from 0 to 1, leader changed from <none> to e16e42524090435eb7561f44a8815878 (127.27.255.65). New cstate: current_term: 1 leader_uuid: "e16e42524090435eb7561f44a8815878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: HEALTHY } } }
I20260505 20:18:11.493451 30244 catalog_manager.cc:5671] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecce4a805d624a32b52d29f1cb860010 (127.27.255.66). New cstate: current_term: 1 leader_uuid: "ecce4a805d624a32b52d29f1cb860010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } } }
I20260505 20:18:11.586846 30564 heartbeater.cc:499] Master 127.27.255.124:41987 was elected leader, sending a full tablet report...
I20260505 20:18:11.587777 30486 heartbeater.cc:499] Master 127.27.255.124:41987 was elected leader, sending a full tablet report...
I20260505 20:18:12.276903 30382 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc"
dest_uuid: "e16e42524090435eb7561f44a8815878"
from {username='slave'} at 127.0.0.1:55638
I20260505 20:18:12.277640 30382 raft_consensus.cc:591] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 1 LEADER]: Received request to step down
I20260505 20:18:12.277930 30382 raft_consensus.cc:3055] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 1 LEADER]: Stepping down as leader of term 1
I20260505 20:18:12.278169 30382 raft_consensus.cc:740] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e16e42524090435eb7561f44a8815878, State: Running, Role: LEADER
I20260505 20:18:12.278769 30382 consensus_queue.cc:260] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.279729 30382 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:12.433928 30672 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:12.434484 30672 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.436611 30672 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:12.437398 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:12.437831 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:12.438191 30382 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3588319984403a88bdce106eea1e56 in term 2.
I20260505 20:18:12.439229 30502 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878; no voters:
I20260505 20:18:12.442699 30672 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260505 20:18:12.443473 30672 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:12.443838 30672 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:12.454059 30672 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.456192 30672 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 2 election: Requested vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:12.457265 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878"
I20260505 20:18:12.457577 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:12.463783 30382 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3588319984403a88bdce106eea1e56 in term 2.
I20260505 20:18:12.465370 30502 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878; no voters: ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:12.466552 30672 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Leader election won for term 2
I20260505 20:18:12.467680 30672 raft_consensus.cc:697] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 2 LEADER]: Becoming Leader. State: Replica: 7c3588319984403a88bdce106eea1e56, State: Running, Role: LEADER
I20260505 20:18:12.467878 30662 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:12.468339 30662 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.468709 30672 consensus_queue.cc:237] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.470346 30662 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:12.471436 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:12.471731 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:12.472503 30382 raft_consensus.cc:2393] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecce4a805d624a32b52d29f1cb860010 in current term 2: Already voted for candidate 7c3588319984403a88bdce106eea1e56 in this term.
I20260505 20:18:12.474241 30424 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecce4a805d624a32b52d29f1cb860010; no voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878
I20260505 20:18:12.475359 30662 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:12.478988 30242 catalog_manager.cc:5671] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 reported cstate change: term changed from 1 to 2, leader changed from e16e42524090435eb7561f44a8815878 (127.27.255.65) to 7c3588319984403a88bdce106eea1e56 (127.27.255.67). New cstate: current_term: 2 leader_uuid: "7c3588319984403a88bdce106eea1e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: HEALTHY } } }
I20260505 20:18:12.483001 30662 raft_consensus.cc:2749] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260505 20:18:12.511076 30460 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1"
dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
from {username='slave'} at 127.0.0.1:53422
I20260505 20:18:12.511768 30460 raft_consensus.cc:591] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 1 LEADER]: Received request to step down
I20260505 20:18:12.512132 30460 raft_consensus.cc:3055] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 1 LEADER]: Stepping down as leader of term 1
I20260505 20:18:12.512439 30460 raft_consensus.cc:740] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecce4a805d624a32b52d29f1cb860010, State: Running, Role: LEADER
I20260505 20:18:12.513298 30460 consensus_queue.cc:260] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.516180 30460 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:12.589704 30460 raft_consensus.cc:1275] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Refusing update from remote peer 7c3588319984403a88bdce106eea1e56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260505 20:18:12.591113 30672 consensus_queue.cc:1048] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260505 20:18:12.592022 30382 raft_consensus.cc:1275] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Refusing update from remote peer 7c3588319984403a88bdce106eea1e56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260505 20:18:12.593331 30672 consensus_queue.cc:1048] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260505 20:18:12.634351 30656 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:12.634815 30656 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.636807 30656 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:12.637732 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:12.637928 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:12.638511 30460 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 2.
I20260505 20:18:12.639698 30345 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters: 7c3588319984403a88bdce106eea1e56
I20260505 20:18:12.640636 30656 raft_consensus.cc:2804] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260505 20:18:12.640995 30656 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:12.641403 30656 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:12.646196 30656 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.647747 30656 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 2 election: Requested vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:12.648650 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:12.648825 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56"
I20260505 20:18:12.649459 30539 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:12.653478 30460 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 2.
I20260505 20:18:12.654994 30345 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:12.655157 30539 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 2.
I20260505 20:18:12.655849 30673 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:12.655877 30656 raft_consensus.cc:2804] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Leader election won for term 2
I20260505 20:18:12.656497 30673 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.657189 30656 raft_consensus.cc:697] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 2 LEADER]: Becoming Leader. State: Replica: e16e42524090435eb7561f44a8815878, State: Running, Role: LEADER
I20260505 20:18:12.658051 30656 consensus_queue.cc:237] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.659741 30673 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:12.659751 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:12.661226 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:12.661875 30460 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3588319984403a88bdce106eea1e56 in term 2.
I20260505 20:18:12.662963 30502 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, ecce4a805d624a32b52d29f1cb860010; no voters: e16e42524090435eb7561f44a8815878
I20260505 20:18:12.664183 30673 raft_consensus.cc:2804] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260505 20:18:12.664600 30673 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:12.665002 30673 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:18:12.667079 30245 catalog_manager.cc:5671] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 reported cstate change: term changed from 1 to 2, leader changed from ecce4a805d624a32b52d29f1cb860010 (127.27.255.66) to e16e42524090435eb7561f44a8815878 (127.27.255.65). New cstate: current_term: 2 leader_uuid: "e16e42524090435eb7561f44a8815878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20260505 20:18:12.671548 30673 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.674034 30673 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 election: Requested vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:12.674295 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878"
I20260505 20:18:12.676940 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:12.677565 30460 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:18:12.684479 30460 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3588319984403a88bdce106eea1e56 in term 3.
I20260505 20:18:12.686028 30502 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, ecce4a805d624a32b52d29f1cb860010; no voters: e16e42524090435eb7561f44a8815878
I20260505 20:18:12.686781 30673 raft_consensus.cc:2804] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Leader election won for term 3
I20260505 20:18:12.687188 30673 raft_consensus.cc:697] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 3 LEADER]: Becoming Leader. State: Replica: 7c3588319984403a88bdce106eea1e56, State: Running, Role: LEADER
I20260505 20:18:12.688045 30673 consensus_queue.cc:237] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.695533 30245 catalog_manager.cc:5671] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 reported cstate change: term changed from 2 to 3, leader changed from e16e42524090435eb7561f44a8815878 (127.27.255.65) to 7c3588319984403a88bdce106eea1e56 (127.27.255.67). New cstate: current_term: 3 leader_uuid: "7c3588319984403a88bdce106eea1e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: HEALTHY } } }
I20260505 20:18:12.722558 30460 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0"
dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
from {username='slave'} at 127.0.0.1:53422
I20260505 20:18:12.723256 30460 raft_consensus.cc:591] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 1 LEADER]: Received request to step down
I20260505 20:18:12.723677 30460 raft_consensus.cc:3055] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 1 LEADER]: Stepping down as leader of term 1
I20260505 20:18:12.724042 30460 raft_consensus.cc:740] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecce4a805d624a32b52d29f1cb860010, State: Running, Role: LEADER
I20260505 20:18:12.727598 30460 consensus_queue.cc:260] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.734493 30460 raft_consensus.cc:3060] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:12.752372 30460 raft_consensus.cc:1240] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 3 FOLLOWER]: Rejecting Update request from peer e16e42524090435eb7561f44a8815878 for earlier term 2. Current term is 3. Ops: []
I20260505 20:18:12.753930 30656 consensus_queue.cc:1059] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260505 20:18:12.754812 30685 raft_consensus.cc:3055] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 2 LEADER]: Stepping down as leader of term 2
I20260505 20:18:12.755190 30685 raft_consensus.cc:740] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e16e42524090435eb7561f44a8815878, State: Running, Role: LEADER
I20260505 20:18:12.756165 30685 consensus_queue.cc:260] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.756742 30539 raft_consensus.cc:1240] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 3 LEADER]: Rejecting Update request from peer e16e42524090435eb7561f44a8815878 for earlier term 2. Current term is 3. Ops: []
I20260505 20:18:12.757180 30685 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:18:12.786175 30382 raft_consensus.cc:1275] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Refusing update from remote peer 7c3588319984403a88bdce106eea1e56: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20260505 20:18:12.786657 30382 pending_rounds.cc:85] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878: Aborting all ops after (but not including) 2
I20260505 20:18:12.786962 30382 pending_rounds.cc:107] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878: Aborting uncommitted NO_OP operation due to leader change: 2.3
I20260505 20:18:12.787250 30382 raft_consensus.cc:2889] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260505 20:18:12.788488 30460 raft_consensus.cc:1275] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 3 FOLLOWER]: Refusing update from remote peer 7c3588319984403a88bdce106eea1e56: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260505 20:18:12.789772 30672 consensus_queue.cc:1048] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260505 20:18:12.790665 30673 consensus_queue.cc:1048] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260505 20:18:12.821782 30685 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:12.822508 30685 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.828130 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:12.828859 30460 raft_consensus.cc:2468] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 2.
I20260505 20:18:12.830077 30345 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:12.831072 30673 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:12.831143 30686 raft_consensus.cc:2804] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260505 20:18:12.831766 30685 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:12.831960 30686 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:12.832185 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:12.831928 30673 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.832657 30686 raft_consensus.cc:3060] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:12.833267 30539 raft_consensus.cc:2468] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 1.
I20260505 20:18:12.834185 30673 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), e16e42524090435eb7561f44a8815878 (127.27.255.65:36045)
I20260505 20:18:12.835556 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:12.836292 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:12.836843 30460 raft_consensus.cc:2468] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3588319984403a88bdce106eea1e56 in term 2.
I20260505 20:18:12.837770 30502 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:12.838701 30673 raft_consensus.cc:2804] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260505 20:18:12.839030 30673 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:12.839071 30686 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.839430 30673 raft_consensus.cc:3060] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:12.840186 30382 raft_consensus.cc:2393] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c3588319984403a88bdce106eea1e56 in current term 2: Already voted for candidate e16e42524090435eb7561f44a8815878 in this term.
I20260505 20:18:12.841595 30686 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 2 election: Requested vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:12.842228 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56"
I20260505 20:18:12.845412 30673 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.845847 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:12.847318 30673 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 2 election: Requested vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), e16e42524090435eb7561f44a8815878 (127.27.255.65:36045)
I20260505 20:18:12.847864 30539 raft_consensus.cc:2393] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e16e42524090435eb7561f44a8815878 in current term 2: Already voted for candidate 7c3588319984403a88bdce106eea1e56 in this term.
I20260505 20:18:12.848400 30459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:12.850387 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878"
I20260505 20:18:12.851075 30382 raft_consensus.cc:2393] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c3588319984403a88bdce106eea1e56 in current term 2: Already voted for candidate e16e42524090435eb7561f44a8815878 in this term.
I20260505 20:18:12.852103 30460 raft_consensus.cc:2468] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 2.
I20260505 20:18:12.852355 30502 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:12.853210 30345 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters: 7c3588319984403a88bdce106eea1e56
I20260505 20:18:12.853349 30673 raft_consensus.cc:2749] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260505 20:18:12.854040 30686 raft_consensus.cc:2804] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Leader election won for term 2
I20260505 20:18:12.855067 30686 raft_consensus.cc:697] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 2 LEADER]: Becoming Leader. State: Replica: e16e42524090435eb7561f44a8815878, State: Running, Role: LEADER
I20260505 20:18:12.856720 30686 consensus_queue.cc:237] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.864794 30244 catalog_manager.cc:5671] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 reported cstate change: term changed from 1 to 2, leader changed from ecce4a805d624a32b52d29f1cb860010 (127.27.255.66) to e16e42524090435eb7561f44a8815878 (127.27.255.65). New cstate: current_term: 2 leader_uuid: "e16e42524090435eb7561f44a8815878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20260505 20:18:12.948628 30539 raft_consensus.cc:1275] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Refusing update from remote peer e16e42524090435eb7561f44a8815878: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260505 20:18:12.950922 30460 raft_consensus.cc:1275] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Refusing update from remote peer e16e42524090435eb7561f44a8815878: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260505 20:18:12.952803 30686 consensus_queue.cc:1048] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260505 20:18:12.956218 30539 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1"
dest_uuid: "7c3588319984403a88bdce106eea1e56"
from {username='slave'} at 127.0.0.1:40832
I20260505 20:18:12.956301 30685 consensus_queue.cc:1048] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260505 20:18:12.956979 30539 raft_consensus.cc:591] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 3 LEADER]: Received request to step down
I20260505 20:18:12.957321 30539 raft_consensus.cc:3055] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 3 LEADER]: Stepping down as leader of term 3
I20260505 20:18:12.957715 30539 raft_consensus.cc:740] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7c3588319984403a88bdce106eea1e56, State: Running, Role: LEADER
I20260505 20:18:12.959434 30539 consensus_queue.cc:260] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:12.960796 30539 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Advancing to term 4
I20260505 20:18:13.106513 30656 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7c3588319984403a88bdce106eea1e56)
I20260505 20:18:13.107041 30656 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:13.108847 30656 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:13.109571 30459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:13.110006 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:13.110664 30539 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 4.
I20260505 20:18:13.111685 30346 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878; no voters: ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:13.112574 30656 raft_consensus.cc:2804] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260505 20:18:13.112934 30656 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7c3588319984403a88bdce106eea1e56)
I20260505 20:18:13.113183 30656 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Advancing to term 4
I20260505 20:18:13.117631 30656 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:13.119010 30656 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 4 election: Requested vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:13.120211 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56"
I20260505 20:18:13.120314 30459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:13.128341 30662 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7c3588319984403a88bdce106eea1e56)
I20260505 20:18:13.129137 30662 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:13.131176 30662 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:13.132193 30539 raft_consensus.cc:2468] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 4.
I20260505 20:18:13.132195 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:13.132503 30537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:13.133183 30382 raft_consensus.cc:2393] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecce4a805d624a32b52d29f1cb860010 in current term 4: Already voted for candidate e16e42524090435eb7561f44a8815878 in this term.
I20260505 20:18:13.133292 30537 raft_consensus.cc:2393] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecce4a805d624a32b52d29f1cb860010 in current term 4: Already voted for candidate e16e42524090435eb7561f44a8815878 in this term.
I20260505 20:18:13.133745 30346 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878; no voters: ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:13.134435 30424 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecce4a805d624a32b52d29f1cb860010; no voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878
I20260505 20:18:13.134649 30656 raft_consensus.cc:2804] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Leader election won for term 4
I20260505 20:18:13.135151 30662 raft_consensus.cc:3060] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 3 FOLLOWER]: Advancing to term 4
I20260505 20:18:13.135175 30656 raft_consensus.cc:697] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 4 LEADER]: Becoming Leader. State: Replica: e16e42524090435eb7561f44a8815878, State: Running, Role: LEADER
I20260505 20:18:13.136360 30656 consensus_queue.cc:237] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:13.142292 30662 raft_consensus.cc:2749] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260505 20:18:13.218555 30537 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc"
dest_uuid: "7c3588319984403a88bdce106eea1e56"
from {username='slave'} at 127.0.0.1:40832
I20260505 20:18:13.220227 30537 raft_consensus.cc:591] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 2 LEADER]: Received request to step down
I20260505 20:18:13.221091 30537 raft_consensus.cc:3055] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 2 LEADER]: Stepping down as leader of term 2
I20260505 20:18:13.221719 30244 catalog_manager.cc:5671] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 reported cstate change: term changed from 3 to 4, leader changed from 7c3588319984403a88bdce106eea1e56 (127.27.255.67) to e16e42524090435eb7561f44a8815878 (127.27.255.65). New cstate: current_term: 4 leader_uuid: "e16e42524090435eb7561f44a8815878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20260505 20:18:13.223052 30537 raft_consensus.cc:740] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7c3588319984403a88bdce106eea1e56, State: Running, Role: LEADER
I20260505 20:18:13.228152 30537 consensus_queue.cc:260] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:13.229694 30537 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:18:13.262781 30459 raft_consensus.cc:1275] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Refusing update from remote peer e16e42524090435eb7561f44a8815878: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260505 20:18:13.264787 30537 raft_consensus.cc:1275] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Refusing update from remote peer e16e42524090435eb7561f44a8815878: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260505 20:18:13.265045 30656 consensus_queue.cc:1048] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260505 20:18:13.266870 30656 consensus_queue.cc:1048] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260505 20:18:13.427861 30656 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c3588319984403a88bdce106eea1e56)
I20260505 20:18:13.428395 30656 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:13.428969 30662 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7c3588319984403a88bdce106eea1e56)
I20260505 20:18:13.429698 30662 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:13.430742 30656 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:13.431322 30459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:13.432029 30459 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 2.
I20260505 20:18:13.434118 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:13.434855 30539 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 3.
I20260505 20:18:13.440132 30244 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40804:
table { table_id: "8cddca7fffa74a7983714816b4d4457d" table_name: "test-table" } new_table_name: "NewTableName"
I20260505 20:18:13.441450 30425 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:13.444559 30537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:13.445379 30537 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 3.
I20260505 20:18:13.447227 30691 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260505 20:18:13.447727 30691 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7c3588319984403a88bdce106eea1e56)
I20260505 20:18:13.448169 30691 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:18:13.440135 30345 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:13.449891 30685 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260505 20:18:13.450304 30685 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7c3588319984403a88bdce106eea1e56)
I20260505 20:18:13.451049 30685 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:18:13.455010 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:13.432039 30662 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:13.463201 30691 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:13.469120 30691 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 3 election: Requested vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:13.470291 30537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56"
I20260505 20:18:13.470526 30381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878"
I20260505 20:18:13.475826 30685 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:13.478402 30685 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 3 election: Requested vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:13.481873 30537 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecce4a805d624a32b52d29f1cb860010 in term 3.
I20260505 20:18:13.482340 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:13.484193 30425 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, ecce4a805d624a32b52d29f1cb860010; no voters: e16e42524090435eb7561f44a8815878
I20260505 20:18:13.485071 30460 raft_consensus.cc:2393] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e16e42524090435eb7561f44a8815878 in current term 3: Already voted for candidate ecce4a805d624a32b52d29f1cb860010 in this term.
I20260505 20:18:13.485749 30691 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 3 FOLLOWER]: Leader election won for term 3
I20260505 20:18:13.486320 30691 raft_consensus.cc:697] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 3 LEADER]: Becoming Leader. State: Replica: ecce4a805d624a32b52d29f1cb860010, State: Running, Role: LEADER
I20260505 20:18:13.486472 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56"
I20260505 20:18:13.487300 30539 raft_consensus.cc:2393] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e16e42524090435eb7561f44a8815878 in current term 3: Already voted for candidate ecce4a805d624a32b52d29f1cb860010 in this term.
I20260505 20:18:13.487337 30691 consensus_queue.cc:237] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:13.489346 30346 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e16e42524090435eb7561f44a8815878; no voters: 7c3588319984403a88bdce106eea1e56, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:13.484510 30382 raft_consensus.cc:2393] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ecce4a805d624a32b52d29f1cb860010 in current term 3: Already voted for candidate e16e42524090435eb7561f44a8815878 in this term.
I20260505 20:18:13.496426 30656 raft_consensus.cc:2749] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260505 20:18:13.510814 30244 catalog_manager.cc:5671] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 reported cstate change: term changed from 2 to 3, leader changed from 7c3588319984403a88bdce106eea1e56 (127.27.255.67) to ecce4a805d624a32b52d29f1cb860010 (127.27.255.66). New cstate: current_term: 3 leader_uuid: "ecce4a805d624a32b52d29f1cb860010" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20260505 20:18:13.512989 30244 catalog_manager.cc:5763] TS ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685) does not have the latest schema for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d]). Expected version 1 got 0
I20260505 20:18:13.525593 28669 master.cc:562] Master@127.27.255.124:41987 shutting down...
I20260505 20:18:13.558598 30697 tablet.cc:1724] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260505 20:18:13.575567 30529 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:40836: Illegal state: replica 7c3588319984403a88bdce106eea1e56 is not leader of this config: current role FOLLOWER
I20260505 20:18:13.584324 30700 tablet.cc:1724] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260505 20:18:13.584216 30697 tablet.cc:1724] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260505 20:18:13.585041 30538 raft_consensus.cc:1275] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Refusing update from remote peer ecce4a805d624a32b52d29f1cb860010: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260505 20:18:13.586549 30382 raft_consensus.cc:1275] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Refusing update from remote peer ecce4a805d624a32b52d29f1cb860010: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260505 20:18:13.597750 30691 consensus_queue.cc:1048] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260505 20:18:13.588277 30230 catalog_manager.cc:5081] TS 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263): alter failed for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d]): Illegal state: replica 7c3588319984403a88bdce106eea1e56 is not leader of this config: current role FOLLOWER
I20260505 20:18:13.588127 30701 tablet.cc:1724] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260505 20:18:13.611873 30691 consensus_queue.cc:1048] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260505 20:18:13.596115 30702 tablet.cc:1724] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260505 20:18:13.636716 30698 tablet.cc:1724] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260505 20:18:13.747745 30698 tablet.cc:1724] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260505 20:18:13.749138 30702 tablet.cc:1724] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260505 20:18:13.749848 30697 tablet.cc:1724] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260505 20:18:13.767777 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 2 LEADER]: Raft consensus shutting down.
I20260505 20:18:13.768898 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:13.769399 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: stopping tablet replica
I20260505 20:18:13.802014 28669 master.cc:584] Master@127.27.255.124:41987 shutdown complete.
I20260505 20:18:13.922092 30709 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6f405746a59b4a56ad59181504704852)
I20260505 20:18:13.922504 30709 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:13.923977 30709 leader_election.cc:290] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:13.925275 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce" is_pre_election: true
W20260505 20:18:13.930429 30100 leader_election.cc:336] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987): Network error: Client connection negotiation failed: client connection to 127.27.255.124:41987: connect: Connection refused (error 111)
I20260505 20:18:13.930789 30100 leader_election.cc:304] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843; no voters: 6f405746a59b4a56ad59181504704852, ef83e6b036a84a31b76b0df6a080dbce
I20260505 20:18:13.931457 30709 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260505 20:18:13.945429 30712 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6f405746a59b4a56ad59181504704852)
I20260505 20:18:13.945987 30712 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:13.947551 30712 leader_election.cc:290] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:13.948931 30124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef83e6b036a84a31b76b0df6a080dbce" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843" is_pre_election: true
I20260505 20:18:13.949558 30124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef83e6b036a84a31b76b0df6a080dbce in term 2.
I20260505 20:18:13.950810 30165 leader_election.cc:304] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843, ef83e6b036a84a31b76b0df6a080dbce; no voters:
I20260505 20:18:13.951511 30712 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260505 20:18:13.951895 30712 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6f405746a59b4a56ad59181504704852)
I20260505 20:18:13.952337 30712 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 2 FOLLOWER]: Advancing to term 3
W20260505 20:18:13.955013 30165 leader_election.cc:336] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987): Network error: Client connection negotiation failed: client connection to 127.27.255.124:41987: connect: Connection refused (error 111)
I20260505 20:18:13.959246 30712 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:13.961802 30712 leader_election.cc:290] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 3 election: Requested vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:13.962719 30124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef83e6b036a84a31b76b0df6a080dbce" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843"
I20260505 20:18:13.963335 30124 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 2 FOLLOWER]: Advancing to term 3
W20260505 20:18:13.965873 30165 leader_election.cc:336] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987): Network error: Client connection negotiation failed: client connection to 127.27.255.124:41987: connect: Connection refused (error 111)
I20260505 20:18:13.968979 30124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef83e6b036a84a31b76b0df6a080dbce in term 3.
I20260505 20:18:13.970237 30165 leader_election.cc:304] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843, ef83e6b036a84a31b76b0df6a080dbce; no voters: 6f405746a59b4a56ad59181504704852
I20260505 20:18:13.972939 30712 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 3 FOLLOWER]: Leader election won for term 3
I20260505 20:18:13.974598 30712 raft_consensus.cc:697] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 3 LEADER]: Becoming Leader. State: Replica: ef83e6b036a84a31b76b0df6a080dbce, State: Running, Role: LEADER
I20260505 20:18:13.975598 30712 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:13.983573 30715 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef83e6b036a84a31b76b0df6a080dbce. Latest consensus state: current_term: 3 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:13.984431 30715 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:13.986119 30716 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:13.993362 30716 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=8cddca7fffa74a7983714816b4d4457d]
I20260505 20:18:13.998298 30716 tablet_loader.cc:96] loaded metadata for tablet 10ea1d60fc224e79a768017a09747cc0 (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:14.000102 30716 tablet_loader.cc:96] loaded metadata for tablet 6f497a2e79934ecc9e9ab47b7d98a9d1 (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:14.001199 30716 tablet_loader.cc:96] loaded metadata for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:14.002477 30716 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:14.005584 30716 catalog_manager.cc:1269] Loaded cluster ID: 1cd94111d53d4222a299fb808db64b51
I20260505 20:18:14.005852 30716 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:14.010922 30716 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:14.014230 30716 catalog_manager.cc:6055] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce: Loaded TSK: 0
I20260505 20:18:14.015165 30716 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:14.020161 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:14.029138 30718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:14.028944 30717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:14.033113 30720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:14.033896 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:14.036815 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:14.037140 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:14.037542 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012294037429 us; error 0 us; skew 500 ppm
I20260505 20:18:14.038733 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:14.041868 28669 webserver.cc:492] Webserver started at http://127.27.255.124:35143/ using document root <none> and password file <none>
I20260505 20:18:14.042454 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:14.042666 28669 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260505 20:18:14.047547 28669 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260505 20:18:14.052289 30725 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:14.053772 28669 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260505 20:18:14.054247 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-2-root
uuid: "6f405746a59b4a56ad59181504704852"
format_stamp: "Formatted at 2026-05-05 20:18:09 on dist-test-slave-dnnj"
I20260505 20:18:14.054673 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:14.068331 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20260505 20:18:14.068552 30165 consensus_peers.cc:597] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce -> Peer 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987): Couldn't send request to peer 6f405746a59b4a56ad59181504704852. Status: Network error: Client connection negotiation failed: client connection to 127.27.255.124:41987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260505 20:18:14.070093 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:14.089938 30124 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 3 FOLLOWER]: Refusing update from remote peer ef83e6b036a84a31b76b0df6a080dbce: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20260505 20:18:14.092206 30715 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260505 20:18:14.097759 30709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef83e6b036a84a31b76b0df6a080dbce. Latest consensus state: current_term: 3 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:14.098817 30709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:14.102317 30715 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:14.103494 30715 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:14.106803 30709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:14.107931 30709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:14.137276 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.124:41987
I20260505 20:18:14.137377 30779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.124:41987 every 8 connection(s)
I20260505 20:18:14.147953 30780 sys_catalog.cc:263] Verifying existing consensus state
I20260505 20:18:14.153247 30780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: Bootstrap starting.
I20260505 20:18:14.190088 30565 heartbeater.cc:499] Master 127.27.255.125:45869 was elected leader, sending a full tablet report...
I20260505 20:18:14.194010 30487 heartbeater.cc:499] Master 127.27.255.125:45869 was elected leader, sending a full tablet report...
I20260505 20:18:14.195122 30486 heartbeater.cc:344] Connected to a master server at 127.27.255.124:41987
I20260505 20:18:14.197863 30409 heartbeater.cc:499] Master 127.27.255.125:45869 was elected leader, sending a full tablet report...
I20260505 20:18:14.212435 30564 heartbeater.cc:344] Connected to a master server at 127.27.255.124:41987
I20260505 20:18:14.460970 30786 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:14.462049 30786 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:14.466730 30786 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), e16e42524090435eb7561f44a8815878 (127.27.255.65:36045)
I20260505 20:18:14.467444 30786 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:14.468019 30786 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:14.471634 30786 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:14.474150 30785 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:14.477170 30785 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:14.480011 30785 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:14.480815 30785 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
W20260505 20:18:14.474220 30425 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.255.67:40263, user_credentials={real_user=slave}} blocked reactor thread for 116448us
I20260505 20:18:14.624192 30787 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:14.721429 30785 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
W20260505 20:18:14.683065 30502 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.27.255.66:42685 timed out after 0.217s (SENT) [suppressed 16 similar messages]
I20260505 20:18:14.895707 30787 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:14.897104 30785 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:15.416394 30787 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:14.944758 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:15.414376 30179 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49958 (request call id 57) took 1212 ms. Trace:
W20260505 20:18:15.418254 30460 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 24) took 523 ms (client timeout 220 ms). Trace:
I20260505 20:18:15.418133 30179 rpcz_store.cc:276] 0505 20:18:14.201559 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:14.201722 (+ 163us) service_pool.cc:224] Handling call
0505 20:18:15.414197 (+1212475us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1214,"reported_tablets":3}
W20260505 20:18:15.419070 30460 rpcz_store.cc:269] 0505 20:18:14.894780 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:14.944444 (+ 49664us) service_pool.cc:224] Handling call
0505 20:18:15.418143 (+473699us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260505 20:18:14.948067 30424 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.004s sys 0.008s
W20260505 20:18:15.420643 30424 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.000s user 0.005s sys 0.008s
I20260505 20:18:14.952592 30459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:14.954444 30381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
W20260505 20:18:15.423627 30381 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55616 (request call id 23) took 469 ms (client timeout 217 ms). Trace:
W20260505 20:18:15.424088 30381 rpcz_store.cc:269] 0505 20:18:14.953948 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:14.954110 (+ 162us) service_pool.cc:224] Handling call
0505 20:18:15.423514 (+469404us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260505 20:18:15.116052 30383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
W20260505 20:18:15.425586 30383 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55616 (request call id 24) took 471 ms (client timeout 220 ms). Trace:
I20260505 20:18:14.837050 30656 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:14.937906 30709 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:14.900854 30780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=7 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 20:18:15.432802 30780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: Bootstrap complete.
I20260505 20:18:15.405779 30408 heartbeater.cc:344] Connected to a master server at 127.27.255.124:41987
I20260505 20:18:15.220481 30180 catalog_manager.cc:6216] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce: NewTableName [id=8cddca7fffa74a7983714816b4d4457d] alter complete (version 1)
I20260505 20:18:15.413508 30178 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49964 (request call id 54) took 1208 ms. Trace:
W20260505 20:18:15.425973 30502 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.255.66:42685, user_credentials={real_user=slave}} blocked reactor thread for 743061us
I20260505 20:18:15.426159 30380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:15.435081 30180 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49976 (request call id 54) took 1241 ms. Trace:
I20260505 20:18:15.434926 30178 rpcz_store.cc:276] 0505 20:18:14.205011 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:14.205259 (+ 248us) service_pool.cc:224] Handling call
0505 20:18:15.413386 (+1208127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":195738,"reported_tablets":3}
W20260505 20:18:15.423610 30459 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53396 (request call id 23) took 529 ms (client timeout 217 ms). Trace:
I20260505 20:18:15.428782 30656 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:15.436175 30780 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:15.437070 30780 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f405746a59b4a56ad59181504704852, State: Initialized, Role: FOLLOWER
I20260505 20:18:15.438071 30780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:15.435469 30180 rpcz_store.cc:276] 0505 20:18:14.193825 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:14.193993 (+ 168us) service_pool.cc:224] Handling call
0505 20:18:15.435021 (+1241028us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 20:18:14.205349 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0505 20:18:14.205950 (+ 601us) write_op.cc:432] Acquiring schema lock in shared mode
0505 20:18:14.205985 (+ 35us) write_op.cc:435] Acquired schema lock
0505 20:18:14.206002 (+ 17us) tablet.cc:662] Decoding operations
0505 20:18:14.206334 (+ 332us) write_op.cc:620] Acquiring the partition lock for write op
0505 20:18:14.206394 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0505 20:18:14.206421 (+ 27us) tablet.cc:685] Acquiring locks for 1 operations
0505 20:18:14.206628 (+ 207us) tablet.cc:701] Row locks acquired
0505 20:18:14.206648 (+ 20us) write_op.cc:260] PREPARE: finished
0505 20:18:14.206786 (+ 138us) write_op.cc:270] Start()
0505 20:18:14.206906 (+ 120us) write_op.cc:276] Timestamp: P: 1778012294206745 usec, L: 0
0505 20:18:14.206936 (+ 30us) op_driver.cc:348] REPLICATION: starting
0505 20:18:14.207731 (+ 795us) log.cc:844] Serialized 331 byte log entry
0505 20:18:14.942603 (+734872us) op_driver.cc:464] REPLICATION: finished
0505 20:18:14.943814 (+ 1211us) write_op.cc:301] APPLY: starting
0505 20:18:14.943890 (+ 76us) tablet.cc:1367] starting BulkCheckPresence
0505 20:18:14.943993 (+ 103us) tablet.cc:1370] finished BulkCheckPresence
0505 20:18:14.944106 (+ 113us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 20:18:14.944306 (+ 200us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 20:18:14.944464 (+ 158us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0505 20:18:14.944517 (+ 53us) write_op.cc:312] APPLY: finished
0505 20:18:14.944704 (+ 187us) log.cc:844] Serialized 22 byte log entry
0505 20:18:14.944865 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0505 20:18:14.944933 (+ 68us) write_op.cc:454] Released schema lock
0505 20:18:14.944968 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0505 20:18:14.945295 (+ 327us) write_op.cc:489] Releasing partition, row and schema locks
0505 20:18:14.945326 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":189383,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":988,"apply.run_cpu_time_us":1464,"apply.run_wall_time_us":1480,"num_ops":1,"prepare.queue_time_us":126,"prepare.run_cpu_time_us":2809,"prepare.run_wall_time_us":2826,"raft.queue_time_us":1634,"raft.run_cpu_time_us":823,"raft.run_wall_time_us":822,"replication_time_us":735502,"thread_start_us":634,"threads_started":1}]]}
I20260505 20:18:15.445339 30780 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: configured and running, proceeding with master startup.
W20260505 20:18:15.426029 30383 rpcz_store.cc:269] 0505 20:18:14.954317 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:15.115609 (+161292us) service_pool.cc:224] Handling call
0505 20:18:15.425481 (+309872us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260505 20:18:15.426949 30379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:15.429427 30709 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:15.448935 30709 leader_election.cc:290] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:15.449375 30796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:15.450325 30796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:15.450749 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce" is_pre_election: true
W20260505 20:18:15.445664 30459 rpcz_store.cc:269] 0505 20:18:14.894410 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:14.952259 (+ 57849us) service_pool.cc:224] Handling call
0505 20:18:15.423504 (+471245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260505 20:18:15.455767 30656 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:15.460840 30537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:15.461659 30537 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 3.
I20260505 20:18:15.464818 30459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:15.466657 30112 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34392 (request call id 57) took 1040 ms. Trace:
I20260505 20:18:15.466885 30346 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878; no voters:
I20260505 20:18:15.467311 30112 rpcz_store.cc:276] 0505 20:18:14.426267 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:14.903341 (+477074us) service_pool.cc:224] Handling call
0505 20:18:15.465309 (+561968us) spinlock_profiling.cc:243] Waited 555 ms on lock 0x7b540013f454. stack: 00007fe78bbcdc83 00007fe78bbcdb80 00007fe78b51b574 00000000003bff74 00000000003bff09 00007fe79256e470 00007fe79260d31a 00007fe794a4246c 00007fe794b3c3ee 00007fe7930264a4 00007fe793026432 00007fe793026361 00007fe7930262dc 00007fe793025592 00007fe7907d4e4c 00007fe7907d4286
0505 20:18:15.466446 (+ 1137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1276565376}
I20260505 20:18:15.472453 30793 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260505 20:18:15.472909 30793 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:15.473169 30755 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 2 FOLLOWER]: Advancing to term 3
I20260505 20:18:15.497186 30754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "6f405746a59b4a56ad59181504704852" is_pre_election: true
I20260505 20:18:15.476275 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
W20260505 20:18:15.488260 30425 leader_election.cc:336] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263): Timed out: RequestConsensusVote RPC to 127.27.255.67:40263 timed out after 0.293s (SENT)
I20260505 20:18:15.476872 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:15.473284 30793 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 3 FOLLOWER]: Advancing to term 4
I20260505 20:18:15.504817 30537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:15.513798 30538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:15.516667 30425 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecce4a805d624a32b52d29f1cb860010; no voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878
W20260505 20:18:15.519572 30425 leader_election.cc:336] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263): Timed out: RequestConsensusVote RPC to 127.27.255.67:40263 timed out after 0.297s (SENT)
I20260505 20:18:15.520807 30425 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecce4a805d624a32b52d29f1cb860010; no voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878
I20260505 20:18:15.521334 30785 raft_consensus.cc:2749] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260505 20:18:15.504319 30100 leader_election.cc:304] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843; no voters: 6f405746a59b4a56ad59181504704852, ef83e6b036a84a31b76b0df6a080dbce
I20260505 20:18:15.526051 30709 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260505 20:18:15.526765 30785 raft_consensus.cc:2749] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260505 20:18:15.527020 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:15.528997 30538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:15.530354 30537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:15.531075 30502 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:15.532625 30787 raft_consensus.cc:2749] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260505 20:18:15.534854 30379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:15.535182 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:15.537395 30502 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:15.538276 30787 raft_consensus.cc:2749] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260505 20:18:15.539695 30793 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:15.540623 30382 raft_consensus.cc:1240] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Rejecting Update request from peer ecce4a805d624a32b52d29f1cb860010 for earlier term 3. Current term is 4. Ops: []
I20260505 20:18:15.541664 30793 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 4 election: Requested vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:15.542321 30459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:15.542554 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:15.545704 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56"
I20260505 20:18:15.550418 30346 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e16e42524090435eb7561f44a8815878; no voters: 7c3588319984403a88bdce106eea1e56, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:15.551852 30793 raft_consensus.cc:2749] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260505 20:18:15.556457 30502 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:15.556747 30785 consensus_queue.cc:1059] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260505 20:18:15.557874 30808 raft_consensus.cc:3055] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 3 LEADER]: Stepping down as leader of term 3
I20260505 20:18:15.558287 30808 raft_consensus.cc:740] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ecce4a805d624a32b52d29f1cb860010, State: Running, Role: LEADER
I20260505 20:18:15.558756 30796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef83e6b036a84a31b76b0df6a080dbce. Latest consensus state: current_term: 3 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:15.559587 30796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:15.559207 30808 consensus_queue.cc:260] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:15.560010 30813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:15.560635 30808 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 3 FOLLOWER]: Advancing to term 4
I20260505 20:18:15.560849 30813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:15.561676 30787 raft_consensus.cc:2749] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260505 20:18:15.566162 30755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40986 (request call id 0) took 1395 ms. Trace:
I20260505 20:18:15.566601 30755 rpcz_store.cc:276] 0505 20:18:14.170638 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:14.170792 (+ 154us) service_pool.cc:224] Handling call
0505 20:18:15.472791 (+1301999us) raft_consensus.cc:1471] Updating replica for 1 ops
0505 20:18:15.533736 (+ 60945us) raft_consensus.cc:1534] Early marking committed up to index 0
0505 20:18:15.533773 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0505 20:18:15.534235 (+ 462us) log.cc:844] Serialized 26 byte log entry
0505 20:18:15.544225 (+ 9990us) raft_consensus.cc:1645] Marking committed up to 17
0505 20:18:15.548990 (+ 4765us) log.cc:844] Serialized 14 byte log entry
0505 20:18:15.556571 (+ 7581us) raft_consensus.cc:1697] Filling consensus response to leader.
0505 20:18:15.556667 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0505 20:18:15.558664 (+ 1997us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b54003912d4. stack: 00007fe78bbcdc83 00007fe78bbcdb80 00007fe78b51b574 00000000003bff74 00000000003bff09 00007fe79256e470 00007fe79260d31a 00007fe794b748ac 00007fe794b7cebc 00007fe794b7ce68 00007fe794b7cde8 00007fe794b7cda3 00007fe794b7c04c 00007fe792a4aad6 00007fe792a4aa68 00007fe79262397a
0505 20:18:15.559919 (+ 1255us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b54003912d4. stack: 00007fe78bbcdc83 00007fe78bbcdb80 00007fe78b51b574 00000000003bff74 00000000003bff09 00007fe79256e470 00007fe79260d31a 00007fe794b748ac 00007fe794b7cebc 00007fe794b7ce68 00007fe794b7cde8 00007fe794b7cda3 00007fe794b7c04c 00007fe792a4aad6 00007fe792a4aa68 00007fe79262397a
0505 20:18:15.565943 (+ 6024us) raft_consensus.cc:1685] finished
0505 20:18:15.565968 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0505 20:18:15.566091 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":4257,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":7245,"raft.run_cpu_time_us":6046,"raft.run_wall_time_us":32396,"spinlock_wait_cycles":57687680,"thread_start_us":21123,"threads_started":3,"wal-append.queue_time_us":8323}
I20260505 20:18:15.570235 30816 catalog_manager.cc:1269] Loaded cluster ID: 1cd94111d53d4222a299fb808db64b51
I20260505 20:18:15.570562 30816 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: loading cluster ID for follower catalog manager: success
I20260505 20:18:15.571162 30804 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:15.571888 30804 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:15.584338 30816 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: acquiring CA information for follower catalog manager: success
I20260505 20:18:15.590512 30816 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260505 20:18:15.590291 30745 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e16e42524090435eb7561f44a8815878" instance_seqno: 1778012289915229) as {username='slave'} at 127.0.0.1:41016; Asking this server to re-register.
I20260505 20:18:15.592471 30408 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:15.595566 30745 ts_manager.cc:194] Registered new tserver with Master: e16e42524090435eb7561f44a8815878 (127.27.255.65:36045)
I20260505 20:18:15.631294 30745 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" instance_seqno: 1778012290089402) as {username='slave'} at 127.0.0.1:40990; Asking this server to re-register.
I20260505 20:18:15.633029 30486 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:15.636605 30745 ts_manager.cc:194] Registered new tserver with Master: ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:15.642856 30745 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "7c3588319984403a88bdce106eea1e56" instance_seqno: 1778012290285033) as {username='slave'} at 127.0.0.1:41002; Asking this server to re-register.
I20260505 20:18:15.644470 30564 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:15.647815 30745 ts_manager.cc:194] Registered new tserver with Master: 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:15.648243 28669 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 1.507333966s
I20260505 20:18:15.760334 30787 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:15.760841 30787 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:15.762975 30787 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:15.763983 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:15.764202 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:15.764631 30460 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c3588319984403a88bdce106eea1e56 in term 4.
I20260505 20:18:15.764981 30382 raft_consensus.cc:2393] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c3588319984403a88bdce106eea1e56 in current term 4: Already voted for candidate e16e42524090435eb7561f44a8815878 in this term.
I20260505 20:18:15.765787 30502 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, ecce4a805d624a32b52d29f1cb860010; no voters:
I20260505 20:18:15.766525 30787 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260505 20:18:15.766957 30787 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ecce4a805d624a32b52d29f1cb860010)
I20260505 20:18:15.767300 30787 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 3 FOLLOWER]: Advancing to term 4
I20260505 20:18:15.775928 30787 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:15.778199 30787 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 4 election: Requested vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:15.778934 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878"
I20260505 20:18:15.779204 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:15.779873 30382 raft_consensus.cc:2393] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c3588319984403a88bdce106eea1e56 in current term 4: Already voted for candidate e16e42524090435eb7561f44a8815878 in this term.
I20260505 20:18:15.785753 30460 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c3588319984403a88bdce106eea1e56 in term 4.
I20260505 20:18:15.786958 30502 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7c3588319984403a88bdce106eea1e56, ecce4a805d624a32b52d29f1cb860010; no voters: e16e42524090435eb7561f44a8815878
I20260505 20:18:15.787873 30787 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Leader election won for term 4
I20260505 20:18:15.788918 30787 raft_consensus.cc:697] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 4 LEADER]: Becoming Leader. State: Replica: 7c3588319984403a88bdce106eea1e56, State: Running, Role: LEADER
I20260505 20:18:15.789903 30787 consensus_queue.cc:237] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:15.801132 30178 catalog_manager.cc:5671] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 reported cstate change: term changed from 3 to 4, leader changed from ecce4a805d624a32b52d29f1cb860010 (127.27.255.66) to 7c3588319984403a88bdce106eea1e56 (127.27.255.67). New cstate: current_term: 4 leader_uuid: "7c3588319984403a88bdce106eea1e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: HEALTHY } } }
I20260505 20:18:15.851660 30793 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:15.852202 30793 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:15.853721 30793 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:15.854902 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:15.854902 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:15.855811 30460 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 4.
I20260505 20:18:15.856928 30345 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters: 7c3588319984403a88bdce106eea1e56
I20260505 20:18:15.857756 30793 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260505 20:18:15.858076 30793 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:15.858371 30793 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Advancing to term 5
I20260505 20:18:15.863176 30793 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:15.864830 30793 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 5 election: Requested vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:15.865665 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010"
I20260505 20:18:15.865962 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "e16e42524090435eb7561f44a8815878" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56"
I20260505 20:18:15.866284 30460 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Advancing to term 5
I20260505 20:18:15.873176 30460 raft_consensus.cc:2468] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e16e42524090435eb7561f44a8815878 in term 5.
I20260505 20:18:15.874291 30345 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010; no voters: 7c3588319984403a88bdce106eea1e56
I20260505 20:18:15.875092 30793 raft_consensus.cc:2804] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 5 FOLLOWER]: Leader election won for term 5
I20260505 20:18:15.875569 30793 raft_consensus.cc:697] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 5 LEADER]: Becoming Leader. State: Replica: e16e42524090435eb7561f44a8815878, State: Running, Role: LEADER
I20260505 20:18:15.876662 30793 consensus_queue.cc:237] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:15.885771 30178 catalog_manager.cc:5671] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 reported cstate change: term changed from 4 to 5, leader changed from 7c3588319984403a88bdce106eea1e56 (127.27.255.67) to e16e42524090435eb7561f44a8815878 (127.27.255.65). New cstate: current_term: 5 leader_uuid: "e16e42524090435eb7561f44a8815878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20260505 20:18:15.906452 30382 raft_consensus.cc:1240] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 5 LEADER]: Rejecting Update request from peer 7c3588319984403a88bdce106eea1e56 for earlier term 4. Current term is 5. Ops: []
I20260505 20:18:15.908054 30819 consensus_queue.cc:1059] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260505 20:18:15.908819 30819 consensus_queue.cc:1243] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [LEADER]: Peer e16e42524090435eb7561f44a8815878 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260505 20:18:15.909188 30819 raft_consensus.cc:3055] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 4 LEADER]: Stepping down as leader of term 4
I20260505 20:18:15.909374 30819 raft_consensus.cc:740] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7c3588319984403a88bdce106eea1e56, State: Running, Role: LEADER
I20260505 20:18:15.910133 30819 consensus_queue.cc:260] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:15.911257 30819 raft_consensus.cc:3060] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Advancing to term 5
I20260505 20:18:15.911402 30460 raft_consensus.cc:1240] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 5 FOLLOWER]: Rejecting Update request from peer 7c3588319984403a88bdce106eea1e56 for earlier term 4. Current term is 5. Ops: []
I20260505 20:18:15.980901 30460 raft_consensus.cc:1275] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 5 FOLLOWER]: Refusing update from remote peer e16e42524090435eb7561f44a8815878: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260505 20:18:15.982502 30656 consensus_queue.cc:1048] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260505 20:18:15.996222 30539 raft_consensus.cc:1275] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 5 FOLLOWER]: Refusing update from remote peer e16e42524090435eb7561f44a8815878: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260505 20:18:15.996838 30539 pending_rounds.cc:85] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56: Aborting all ops after (but not including) 5
I20260505 20:18:15.997278 30539 pending_rounds.cc:107] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260505 20:18:15.997591 30539 raft_consensus.cc:2889] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260505 20:18:16.001762 30827 consensus_queue.cc:1048] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
W20260505 20:18:16.134969 30482 debug-util.cc:398] Leaking SignalData structure 0x7b08001caee0 after lost signal to thread 28672
W20260505 20:18:16.136157 30482 debug-util.cc:398] Leaking SignalData structure 0x7b08001cb640 after lost signal to thread 30149
W20260505 20:18:16.137109 30482 debug-util.cc:398] Leaking SignalData structure 0x7b08001cbd00 after lost signal to thread 30214
W20260505 20:18:16.138561 30482 debug-util.cc:398] Leaking SignalData structure 0x7b0800245ac0 after lost signal to thread 30407
W20260505 20:18:16.139693 30482 debug-util.cc:398] Leaking SignalData structure 0x7b08001b7320 after lost signal to thread 30485
W20260505 20:18:16.140616 30482 debug-util.cc:398] Leaking SignalData structure 0x7b08001fd200 after lost signal to thread 30563
I20260505 20:18:22.298826 30837 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:22.299515 30837 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:22.301546 30837 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), e16e42524090435eb7561f44a8815878 (127.27.255.65:36045)
I20260505 20:18:22.302047 30837 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:22.297564 30838 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:22.303416 30838 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:22.305409 30838 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:22.302491 30837 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:22.307534 30837 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:22.309171 30836 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:22.309749 30836 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:22.321084 30836 leader_election.cc:290] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869)
W20260505 20:18:22.632529 30727 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.27.255.126:40673 for RPC RequestConsensusVote timed out after 0.263s (ON_OUTBOUND_QUEUE) [suppressed 5 similar messages]
W20260505 20:18:22.684764 30727 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.255.126:40673, user_credentials={real_user=slave}} blocked reactor thread for 52310.7us
W20260505 20:18:22.774940 30502 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.255.66:42685, user_credentials={real_user=slave}} blocked reactor thread for 132732us
I20260505 20:18:22.912597 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:22.915040 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
W20260505 20:18:23.240025 30382 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55616 (request call id 33) took 323 ms (client timeout 296 ms). Trace:
W20260505 20:18:23.240734 30382 rpcz_store.cc:269] 0505 20:18:22.916438 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:23.239695 (+323257us) service_pool.cc:211] Skipping call since client already timed out
0505 20:18:23.239909 (+ 214us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260505 20:18:23.547863 30178 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49976 (request call id 120) took 1424 ms. Trace:
I20260505 20:18:23.548440 30178 rpcz_store.cc:276] 0505 20:18:22.123021 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 20:18:22.123203 (+ 182us) service_pool.cc:224] Handling call
0505 20:18:23.547773 (+1424570us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":0}
I20260505 20:18:23.612653 30846 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:23.613166 30846 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:23.615307 30846 leader_election.cc:290] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
W20260505 20:18:23.649164 30165 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.255.126:40673, user_credentials={real_user=slave}} blocked reactor thread for 196263us
I20260505 20:18:23.673609 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce" is_pre_election: true
I20260505 20:18:23.722347 30852 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:23.723320 30852 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:23.727161 30852 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:23.730212 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:23.757819 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:23.759645 30379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
W20260505 20:18:23.760048 30727 leader_election.cc:336] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673): Timed out: connection negotiation to 127.27.255.126:40673 for RPC RequestConsensusVote timed out after 0.263s (ON_OUTBOUND_QUEUE)
W20260505 20:18:23.760993 30727 leader_election.cc:336] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869): Timed out: connection negotiation to 127.27.255.125:45869 for RPC RequestConsensusVote timed out after 0.263s (ON_OUTBOUND_QUEUE)
W20260505 20:18:23.761348 30502 leader_election.cc:336] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685): Timed out: RequestConsensusVote RPC to 127.27.255.66:42685 timed out after 0.209s (SENT)
I20260505 20:18:23.761539 30727 leader_election.cc:304] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f405746a59b4a56ad59181504704852; no voters: 2e02ce4bdc754528a079bdb143064843, ef83e6b036a84a31b76b0df6a080dbce
W20260505 20:18:23.762195 30502 leader_election.cc:336] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e16e42524090435eb7561f44a8815878 (127.27.255.65:36045): Timed out: RequestConsensusVote RPC to 127.27.255.65:36045 timed out after 0.209s (SENT)
I20260505 20:18:23.762678 30502 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:23.765002 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:23.767010 30855 raft_consensus.cc:2749] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260505 20:18:23.767647 30502 leader_election.cc:336] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e16e42524090435eb7561f44a8815878 (127.27.255.65:36045): Timed out: RequestConsensusVote RPC to 127.27.255.65:36045 timed out after 0.283s (SENT)
W20260505 20:18:23.769598 30502 leader_election.cc:336] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685): Timed out: RequestConsensusVote RPC to 127.27.255.66:42685 timed out after 0.283s (SENT)
I20260505 20:18:23.770161 30502 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
W20260505 20:18:23.771325 30102 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.853s user 0.016s sys 0.025s
W20260505 20:18:23.771821 30102 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.854s user 0.017s sys 0.026s
W20260505 20:18:23.771857 30502 leader_election.cc:336] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e16e42524090435eb7561f44a8815878 (127.27.255.65:36045): Timed out: RequestConsensusVote RPC to 127.27.255.65:36045 timed out after 0.296s (SENT)
W20260505 20:18:23.772704 30502 leader_election.cc:336] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685): Timed out: RequestConsensusVote RPC to 127.27.255.66:42685 timed out after 0.296s (SENT)
I20260505 20:18:23.773111 30502 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:23.779980 30459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:23.780514 30458 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:23.782341 30457 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:23.784049 30460 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:23.784399 30461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:23.786645 30855 raft_consensus.cc:2749] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260505 20:18:23.788506 30854 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260505 20:18:23.791227 30855 raft_consensus.cc:2749] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260505 20:18:23.797075 30755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "6f405746a59b4a56ad59181504704852" is_pre_election: true
I20260505 20:18:23.797901 30755 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e02ce4bdc754528a079bdb143064843 in term 3.
I20260505 20:18:23.804972 30100 leader_election.cc:304] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843, 6f405746a59b4a56ad59181504704852; no voters: ef83e6b036a84a31b76b0df6a080dbce
I20260505 20:18:23.806879 30846 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260505 20:18:23.807467 30846 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:23.807876 30846 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 3 FOLLOWER]: Advancing to term 4
W20260505 20:18:23.820977 30482 debug-util.cc:398] Leaking SignalData structure 0x7b08000e5460 after lost signal to thread 28672
W20260505 20:18:23.823117 30482 debug-util.cc:398] Leaking SignalData structure 0x7b08001fd300 after lost signal to thread 30149
W20260505 20:18:23.823338 30502 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.255.66:42685, user_credentials={real_user=slave}} blocked reactor thread for 50656.4us
W20260505 20:18:23.824378 30482 debug-util.cc:398] Leaking SignalData structure 0x7b08000f7f00 after lost signal to thread 30214
I20260505 20:18:23.825824 30537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:23.826000 30461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:23.828717 30425 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecce4a805d624a32b52d29f1cb860010; no voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878
I20260505 20:18:23.829372 30846 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
W20260505 20:18:23.831436 30482 debug-util.cc:398] Leaking SignalData structure 0x7b08001358c0 after lost signal to thread 30407
W20260505 20:18:23.833607 30482 debug-util.cc:398] Leaking SignalData structure 0x7b0800218d00 after lost signal to thread 30485
I20260505 20:18:23.831753 30852 raft_consensus.cc:2749] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260505 20:18:23.834765 30482 debug-util.cc:398] Leaking SignalData structure 0x7b0800242d60 after lost signal to thread 30563
W20260505 20:18:23.836422 30482 debug-util.cc:398] Leaking SignalData structure 0x7b08000e2b80 after lost signal to thread 30779
I20260505 20:18:23.838794 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce"
I20260505 20:18:23.841529 30846 leader_election.cc:290] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 4 election: Requested vote from peers ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:23.842211 30755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "6f405746a59b4a56ad59181504704852"
I20260505 20:18:23.843638 30100 leader_election.cc:304] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843; no voters: 6f405746a59b4a56ad59181504704852, ef83e6b036a84a31b76b0df6a080dbce
I20260505 20:18:23.844671 30846 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260505 20:18:23.849539 30125 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 4 FOLLOWER]: Rejecting Update request from peer ef83e6b036a84a31b76b0df6a080dbce for earlier term 3. Current term is 4. Ops: []
I20260505 20:18:23.851753 30853 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 }, Status: INVALID_TERM, Last received: 3.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260505 20:18:23.852936 30853 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 3 LEADER]: Stepping down as leader of term 3
I20260505 20:18:23.853278 30853 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ef83e6b036a84a31b76b0df6a080dbce, State: Running, Role: LEADER
I20260505 20:18:23.854418 30853 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 3.20, Last appended by leader: 20, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:23.856829 30853 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 3 FOLLOWER]: Advancing to term 4
I20260505 20:18:24.011607 30853 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:24.013231 30854 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:24.013804 30854 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:24.012936 30853 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:24.017948 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce" is_pre_election: true
I20260505 20:18:24.018687 30190 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f405746a59b4a56ad59181504704852 in term 4.
I20260505 20:18:24.020390 30125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef83e6b036a84a31b76b0df6a080dbce" candidate_term: 5 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843" is_pre_election: true
I20260505 20:18:24.021059 30125 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef83e6b036a84a31b76b0df6a080dbce in term 4.
I20260505 20:18:24.021163 30853 leader_election.cc:290] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:24.022512 30124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843" is_pre_election: true
I20260505 20:18:24.023348 30124 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f405746a59b4a56ad59181504704852 in current term 4: Already voted for candidate 2e02ce4bdc754528a079bdb143064843 in this term.
I20260505 20:18:24.024243 30755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef83e6b036a84a31b76b0df6a080dbce" candidate_term: 5 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "6f405746a59b4a56ad59181504704852" is_pre_election: true
I20260505 20:18:24.024868 30755 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef83e6b036a84a31b76b0df6a080dbce in term 3.
I20260505 20:18:24.026124 30165 leader_election.cc:304] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f405746a59b4a56ad59181504704852, ef83e6b036a84a31b76b0df6a080dbce; no voters:
I20260505 20:18:24.028450 30853 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260505 20:18:24.028916 30853 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:24.029541 30727 leader_election.cc:304] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f405746a59b4a56ad59181504704852, ef83e6b036a84a31b76b0df6a080dbce; no voters:
I20260505 20:18:24.029989 30853 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 4 FOLLOWER]: Advancing to term 5
I20260505 20:18:24.038394 30857 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260505 20:18:24.038861 30857 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:24.039232 30857 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 3 FOLLOWER]: Advancing to term 4
I20260505 20:18:24.016098 30854 leader_election.cc:290] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869)
I20260505 20:18:24.051599 30853 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:24.055542 30124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef83e6b036a84a31b76b0df6a080dbce" candidate_term: 5 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843"
I20260505 20:18:24.056275 30124 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 4 FOLLOWER]: Advancing to term 5
I20260505 20:18:24.057384 30857 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:24.063057 30125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843"
I20260505 20:18:24.064019 30857 leader_election.cc:290] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 4 election: Requested vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869)
I20260505 20:18:24.064576 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce"
I20260505 20:18:24.065512 30190 raft_consensus.cc:2368] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f405746a59b4a56ad59181504704852 for earlier term 4. Current term is 5.
I20260505 20:18:24.069356 30727 leader_election.cc:400] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 4 election: Vote denied by peer ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f405746a59b4a56ad59181504704852 for earlier term 4. Current term is 5.
I20260505 20:18:24.069747 30727 leader_election.cc:403] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260505 20:18:24.071012 30857 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 4 FOLLOWER]: Advancing to term 5
I20260505 20:18:24.072988 30755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef83e6b036a84a31b76b0df6a080dbce" candidate_term: 5 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "6f405746a59b4a56ad59181504704852"
I20260505 20:18:24.072299 30853 leader_election.cc:290] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 5 election: Requested vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:24.079409 30124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef83e6b036a84a31b76b0df6a080dbce in term 5.
I20260505 20:18:24.085822 30857 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f405746a59b4a56ad59181504704852 for earlier term 4. Current term is 5.
I20260505 20:18:24.085659 30165 leader_election.cc:304] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843, ef83e6b036a84a31b76b0df6a080dbce; no voters:
I20260505 20:18:24.087062 30853 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 5 FOLLOWER]: Leader election won for term 5
I20260505 20:18:24.087601 30853 raft_consensus.cc:697] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 5 LEADER]: Becoming Leader. State: Replica: ef83e6b036a84a31b76b0df6a080dbce, State: Running, Role: LEADER
I20260505 20:18:24.089583 30853 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 3.20, Last appended by leader: 20, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:24.105782 30850 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef83e6b036a84a31b76b0df6a080dbce. Latest consensus state: current_term: 5 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:24.106936 30850 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:24.117733 30755 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef83e6b036a84a31b76b0df6a080dbce in term 5.
I20260505 20:18:24.177655 30755 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 5 FOLLOWER]: Refusing update from remote peer ef83e6b036a84a31b76b0df6a080dbce: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 5 index: 21. (index mismatch)
I20260505 20:18:24.179944 30850 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260505 20:18:24.187307 30857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef83e6b036a84a31b76b0df6a080dbce. Latest consensus state: current_term: 5 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:24.188210 30857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:24.194692 30124 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 5 FOLLOWER]: Refusing update from remote peer ef83e6b036a84a31b76b0df6a080dbce: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 5 index: 21. (index mismatch)
I20260505 20:18:24.196604 30853 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:24.197434 30853 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:24.199465 30857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:24.200250 30857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:24.200791 30853 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260505 20:18:24.208534 30846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef83e6b036a84a31b76b0df6a080dbce. Latest consensus state: current_term: 5 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:24.208408 30866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:24.209318 30846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:24.209421 30866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:24.295712 30850 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "ef83e6b036a84a31b76b0df6a080dbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:24.296626 30850 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:40.038574 30883 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:40.039778 30883 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:40.044037 30883 leader_election.cc:290] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869), 6f405746a59b4a56ad59181504704852 (127.27.255.124:41987)
I20260505 20:18:40.079502 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 6 candidate_status { last_received { term: 5 index: 21 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce" is_pre_election: true
I20260505 20:18:40.142102 30884 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:40.143191 30884 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:40.146587 30884 leader_election.cc:290] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869)
I20260505 20:18:40.148602 30125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 6 candidate_status { last_received { term: 5 index: 21 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843" is_pre_election: true
I20260505 20:18:40.149386 30125 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f405746a59b4a56ad59181504704852 in term 5.
I20260505 20:18:40.151590 30727 leader_election.cc:304] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843, 6f405746a59b4a56ad59181504704852; no voters:
I20260505 20:18:40.152722 30884 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260505 20:18:40.153223 30884 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ef83e6b036a84a31b76b0df6a080dbce)
I20260505 20:18:40.153689 30884 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 5 FOLLOWER]: Advancing to term 6
I20260505 20:18:40.169869 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 6 candidate_status { last_received { term: 5 index: 21 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce" is_pre_election: true
I20260505 20:18:40.170991 30886 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:40.172549 30886 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:40.168670 30884 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:40.176205 30755 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 6 FOLLOWER]: Rejecting Update request from peer ef83e6b036a84a31b76b0df6a080dbce for earlier term 5. Current term is 6. Ops: []
I20260505 20:18:40.176249 30886 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
I20260505 20:18:40.177408 30886 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:40.177964 30886 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:40.180064 30886 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), e16e42524090435eb7561f44a8815878 (127.27.255.65:36045)
I20260505 20:18:40.191784 30755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e02ce4bdc754528a079bdb143064843" candidate_term: 6 candidate_status { last_received { term: 5 index: 21 } } ignore_live_leader: false dest_uuid: "6f405746a59b4a56ad59181504704852" is_pre_election: true
I20260505 20:18:40.193082 30755 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2e02ce4bdc754528a079bdb143064843 in current term 6: Already voted for candidate 6f405746a59b4a56ad59181504704852 in this term.
I20260505 20:18:40.193642 30125 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 6 candidate_status { last_received { term: 5 index: 21 } } ignore_live_leader: false dest_uuid: "2e02ce4bdc754528a079bdb143064843"
I20260505 20:18:40.194312 30125 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 5 FOLLOWER]: Advancing to term 6
I20260505 20:18:40.195335 30884 leader_election.cc:290] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 6 election: Requested vote from peers 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673), ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869)
I20260505 20:18:40.192039 30873 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 }, Status: INVALID_TERM, Last received: 5.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260505 20:18:40.198922 30190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f405746a59b4a56ad59181504704852" candidate_term: 6 candidate_status { last_received { term: 5 index: 21 } } ignore_live_leader: false dest_uuid: "ef83e6b036a84a31b76b0df6a080dbce"
I20260505 20:18:40.199731 30888 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:40.200938 30888 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:40.204632 30888 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), 7c3588319984403a88bdce106eea1e56 (127.27.255.67:40263)
I20260505 20:18:40.205982 30382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:40.200976 30100 leader_election.cc:304] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843; no voters: 6f405746a59b4a56ad59181504704852, ef83e6b036a84a31b76b0df6a080dbce
I20260505 20:18:40.208081 30379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:40.210879 30539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "988a2ee0b4b14b3da256bef5f9dfe7dc" candidate_uuid: "ecce4a805d624a32b52d29f1cb860010" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7c3588319984403a88bdce106eea1e56" is_pre_election: true
I20260505 20:18:40.211205 30457 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f497a2e79934ecc9e9ab47b7d98a9d1" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:40.213894 30383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e16e42524090435eb7561f44a8815878" is_pre_election: true
I20260505 20:18:40.214709 30425 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ecce4a805d624a32b52d29f1cb860010; no voters: 7c3588319984403a88bdce106eea1e56, e16e42524090435eb7561f44a8815878
I20260505 20:18:40.217377 30457 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10ea1d60fc224e79a768017a09747cc0" candidate_uuid: "7c3588319984403a88bdce106eea1e56" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ecce4a805d624a32b52d29f1cb860010" is_pre_election: true
I20260505 20:18:40.215346 30502 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:40.218601 30887 raft_consensus.cc:2749] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260505 20:18:40.220485 30888 raft_consensus.cc:2749] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260505 20:18:40.223234 30502 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:40.227269 30887 raft_consensus.cc:2749] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260505 20:18:40.228812 30125 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f405746a59b4a56ad59181504704852 in term 6.
I20260505 20:18:40.229894 30883 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260505 20:18:40.230891 30124 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 6 FOLLOWER]: Rejecting Update request from peer ef83e6b036a84a31b76b0df6a080dbce for earlier term 5. Current term is 6. Ops: []
I20260505 20:18:40.230568 30727 leader_election.cc:304] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e02ce4bdc754528a079bdb143064843, 6f405746a59b4a56ad59181504704852; no voters: ef83e6b036a84a31b76b0df6a080dbce
I20260505 20:18:40.232010 30884 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 6 FOLLOWER]: Leader election won for term 6
I20260505 20:18:40.242137 30884 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 6 LEADER]: Becoming Leader. State: Replica: 6f405746a59b4a56ad59181504704852, State: Running, Role: LEADER
I20260505 20:18:40.243777 30884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 5.21, Last appended by leader: 21, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:40.249257 30873 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 }, Status: INVALID_TERM, Last received: 5.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260505 20:18:40.253824 30891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f405746a59b4a56ad59181504704852. Latest consensus state: current_term: 6 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:40.255031 30891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:40.257300 30893 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:40.258533 30890 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 5 LEADER]: Stepping down as leader of term 5
I20260505 20:18:40.259112 30890 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ef83e6b036a84a31b76b0df6a080dbce, State: Running, Role: LEADER
I20260505 20:18:40.260596 30890 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 5.21, Last appended by leader: 21, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } }
I20260505 20:18:40.262220 30890 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 5 FOLLOWER]: Advancing to term 6
I20260505 20:18:40.267629 30893 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=8cddca7fffa74a7983714816b4d4457d]
I20260505 20:18:40.274611 30893 tablet_loader.cc:96] loaded metadata for tablet 10ea1d60fc224e79a768017a09747cc0 (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:40.276670 30893 tablet_loader.cc:96] loaded metadata for tablet 6f497a2e79934ecc9e9ab47b7d98a9d1 (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:40.277994 30893 tablet_loader.cc:96] loaded metadata for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:40.278904 30893 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:40.279192 30755 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 6 LEADER]: Rejecting Update request from peer ef83e6b036a84a31b76b0df6a080dbce for earlier term 5. Current term is 6. Ops: []
I20260505 20:18:40.282357 30893 catalog_manager.cc:1269] Loaded cluster ID: 1cd94111d53d4222a299fb808db64b51
I20260505 20:18:40.282724 30893 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:40.287221 30893 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:40.291038 30893 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852: Loaded TSK: 0
I20260505 20:18:40.292078 30893 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:40.345181 30124 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 6 FOLLOWER]: Refusing update from remote peer 6f405746a59b4a56ad59181504704852: Log matching property violated. Preceding OpId in replica: term: 5 index: 21. Preceding OpId from leader: term: 6 index: 22. (index mismatch)
I20260505 20:18:40.347035 30891 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260505 20:18:40.363822 30883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f405746a59b4a56ad59181504704852. Latest consensus state: current_term: 6 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:40.364722 30883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:40.368748 30190 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 6 FOLLOWER]: Refusing update from remote peer 6f405746a59b4a56ad59181504704852: Log matching property violated. Preceding OpId in replica: term: 5 index: 21. Preceding OpId from leader: term: 6 index: 22. (index mismatch)
I20260505 20:18:40.370903 30891 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260505 20:18:40.376907 30883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:40.378258 30883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:40.378295 30891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:40.379768 30891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:40.394248 30873 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:40.394737 30890 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f405746a59b4a56ad59181504704852. Latest consensus state: current_term: 6 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:40.395249 30873 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:40.395562 30890 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [sys.catalog]: This master's current role is: FOLLOWER
I20260505 20:18:40.411604 30408 heartbeater.cc:499] Master 127.27.255.124:41987 was elected leader, sending a full tablet report...
I20260505 20:18:40.432196 30564 heartbeater.cc:499] Master 127.27.255.124:41987 was elected leader, sending a full tablet report...
I20260505 20:18:40.433888 30486 heartbeater.cc:499] Master 127.27.255.124:41987 was elected leader, sending a full tablet report...
I20260505 20:18:40.473174 30891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:40.474123 30891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: LEADER
W20260505 20:18:41.216171 30776 debug-util.cc:398] Leaking SignalData structure 0x7b08000ed360 after lost signal to thread 28672
W20260505 20:18:41.217564 30776 debug-util.cc:398] Leaking SignalData structure 0x7b0800255ca0 after lost signal to thread 30149
W20260505 20:18:41.218670 30776 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6a60 after lost signal to thread 30214
W20260505 20:18:41.220091 30776 debug-util.cc:398] Leaking SignalData structure 0x7b0800087740 after lost signal to thread 30407
W20260505 20:18:41.221145 30776 debug-util.cc:398] Leaking SignalData structure 0x7b08001b6120 after lost signal to thread 30485
W20260505 20:18:41.222184 30776 debug-util.cc:398] Leaking SignalData structure 0x7b08000ffc40 after lost signal to thread 30563
W20260505 20:18:41.223124 30776 debug-util.cc:398] Leaking SignalData structure 0x7b080021f420 after lost signal to thread 30779
W20260505 20:18:41.223516 30776 debug-util.cc:398] Leaking SignalData structure 0x7b08000a8000 after lost signal to thread 30859
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:784: Failure
Expected equality of these values:
live_row_count
Which is: 802
table_info->GetMetrics()->live_row_count->value()
Which is: 0
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260505 20:18:46.140255 30859 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 784, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260505 20:18:46.149660 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:46.158303 30859 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=8cddca7fffa74a7983714816b4d4457d]
I20260505 20:18:46.174425 30859 tablet_loader.cc:96] loaded metadata for tablet 10ea1d60fc224e79a768017a09747cc0 (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:46.176625 30859 tablet_loader.cc:96] loaded metadata for tablet 6f497a2e79934ecc9e9ab47b7d98a9d1 (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:46.178828 30859 tablet_loader.cc:96] loaded metadata for tablet 988a2ee0b4b14b3da256bef5f9dfe7dc (table NewTableName [id=8cddca7fffa74a7983714816b4d4457d])
I20260505 20:18:46.182130 30859 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:46.196460 30859 catalog_manager.cc:1269] Loaded cluster ID: 1cd94111d53d4222a299fb808db64b51
I20260505 20:18:46.196924 30859 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:46.208968 30859 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:46.218253 30859 catalog_manager.cc:6055] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce: Loaded TSK: 0
I20260505 20:18:46.219736 30859 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260505 20:18:46.226159 30859 catalog_manager.cc:1419] Failed waiting for node to catch up after master election: Illegal state: Node ef83e6b036a84a31b76b0df6a080dbce not leader. Raft Consensus state: current_term: 6 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
W20260505 20:18:46.227699 30859 catalog_manager.cc:1419] Failed waiting for node to catch up after master election: Illegal state: Node ef83e6b036a84a31b76b0df6a080dbce not leader. Raft Consensus state: current_term: 6 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:46.232815 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:46.233878 28669 tablet_replica.cc:333] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878: stopping tablet replica
I20260505 20:18:46.234709 28669 raft_consensus.cc:2243] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 4 LEADER]: Raft consensus shutting down.
I20260505 20:18:46.236018 28669 raft_consensus.cc:2272] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P e16e42524090435eb7561f44a8815878 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.241340 28669 tablet_replica.cc:333] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878: stopping tablet replica
I20260505 20:18:46.242581 28669 raft_consensus.cc:2243] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 2 LEADER]: Raft consensus shutting down.
I20260505 20:18:46.243534 28669 raft_consensus.cc:2272] T 10ea1d60fc224e79a768017a09747cc0 P e16e42524090435eb7561f44a8815878 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.246922 28669 tablet_replica.cc:333] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878: stopping tablet replica
I20260505 20:18:46.247478 28669 raft_consensus.cc:2243] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 5 LEADER]: Raft consensus shutting down.
I20260505 20:18:46.248499 28669 raft_consensus.cc:2272] T 988a2ee0b4b14b3da256bef5f9dfe7dc P e16e42524090435eb7561f44a8815878 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.277738 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:46.299451 28669 tablet_server.cc:179] TabletServer@127.27.255.66:0 shutting down...
I20260505 20:18:46.340523 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:46.341538 28669 tablet_replica.cc:333] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010: stopping tablet replica
I20260505 20:18:46.342307 28669 raft_consensus.cc:2243] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:46.343478 28669 raft_consensus.cc:2272] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P ecce4a805d624a32b52d29f1cb860010 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.347568 28669 tablet_replica.cc:333] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010: stopping tablet replica
I20260505 20:18:46.348150 28669 raft_consensus.cc:2243] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:46.348762 28669 raft_consensus.cc:2272] T 10ea1d60fc224e79a768017a09747cc0 P ecce4a805d624a32b52d29f1cb860010 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.351482 28669 tablet_replica.cc:333] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010: stopping tablet replica
I20260505 20:18:46.352018 28669 raft_consensus.cc:2243] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:46.352501 28669 raft_consensus.cc:2272] T 988a2ee0b4b14b3da256bef5f9dfe7dc P ecce4a805d624a32b52d29f1cb860010 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.388981 30906 raft_consensus.cc:493] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:46.389874 30906 raft_consensus.cc:515] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:46.392575 30906 leader_election.cc:290] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685), e16e42524090435eb7561f44a8815878 (127.27.255.65:36045)
W20260505 20:18:46.396723 30502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.255.66:42685: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20260505 20:18:46.399097 30906 raft_consensus.cc:493] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:46.399773 30906 raft_consensus.cc:515] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:46.403407 30906 leader_election.cc:290] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
W20260505 20:18:46.404296 30502 leader_election.cc:336] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685): Network error: Client connection negotiation failed: client connection to 127.27.255.66:42685: connect: Connection refused (error 111)
W20260505 20:18:46.405242 30502 leader_election.cc:336] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e16e42524090435eb7561f44a8815878 (127.27.255.65:36045): Network error: Client connection negotiation failed: client connection to 127.27.255.65:36045: connect: Connection refused (error 111)
I20260505 20:18:46.405635 30502 leader_election.cc:304] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:46.406313 30906 raft_consensus.cc:2749] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260505 20:18:46.408228 30502 leader_election.cc:336] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685): Network error: Client connection negotiation failed: client connection to 127.27.255.66:42685: connect: Connection refused (error 111)
W20260505 20:18:46.408989 30502 leader_election.cc:336] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e16e42524090435eb7561f44a8815878 (127.27.255.65:36045): Network error: Client connection negotiation failed: client connection to 127.27.255.65:36045: connect: Connection refused (error 111)
I20260505 20:18:46.409317 30502 leader_election.cc:304] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:46.410069 30906 raft_consensus.cc:2749] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260505 20:18:46.418304 30906 raft_consensus.cc:493] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e16e42524090435eb7561f44a8815878)
I20260505 20:18:46.419054 30906 raft_consensus.cc:515] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e16e42524090435eb7561f44a8815878" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36045 } } peers { permanent_uuid: "ecce4a805d624a32b52d29f1cb860010" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 42685 } } peers { permanent_uuid: "7c3588319984403a88bdce106eea1e56" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 40263 } }
I20260505 20:18:46.421320 30906 leader_election.cc:290] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e16e42524090435eb7561f44a8815878 (127.27.255.65:36045), ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685)
W20260505 20:18:46.426000 30502 leader_election.cc:336] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e16e42524090435eb7561f44a8815878 (127.27.255.65:36045): Network error: Client connection negotiation failed: client connection to 127.27.255.65:36045: connect: Connection refused (error 111)
W20260505 20:18:46.426754 30502 leader_election.cc:336] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ecce4a805d624a32b52d29f1cb860010 (127.27.255.66:42685): Network error: Client connection negotiation failed: client connection to 127.27.255.66:42685: connect: Connection refused (error 111)
I20260505 20:18:46.427139 30502 leader_election.cc:304] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7c3588319984403a88bdce106eea1e56; no voters: e16e42524090435eb7561f44a8815878, ecce4a805d624a32b52d29f1cb860010
I20260505 20:18:46.427831 30906 raft_consensus.cc:2749] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260505 20:18:46.432112 28669 tablet_server.cc:196] TabletServer@127.27.255.66:0 shutdown complete.
I20260505 20:18:46.455222 28669 tablet_server.cc:179] TabletServer@127.27.255.67:0 shutting down...
I20260505 20:18:46.499929 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:46.500802 28669 tablet_replica.cc:333] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56: stopping tablet replica
I20260505 20:18:46.501891 28669 raft_consensus.cc:2243] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:46.502719 28669 raft_consensus.cc:2272] T 6f497a2e79934ecc9e9ab47b7d98a9d1 P 7c3588319984403a88bdce106eea1e56 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.505725 28669 tablet_replica.cc:333] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56: stopping tablet replica
I20260505 20:18:46.506311 28669 raft_consensus.cc:2243] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:46.506839 28669 raft_consensus.cc:2272] T 10ea1d60fc224e79a768017a09747cc0 P 7c3588319984403a88bdce106eea1e56 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.509440 28669 tablet_replica.cc:333] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56: stopping tablet replica
I20260505 20:18:46.510020 28669 raft_consensus.cc:2243] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:46.510754 28669 raft_consensus.cc:2272] T 988a2ee0b4b14b3da256bef5f9dfe7dc P 7c3588319984403a88bdce106eea1e56 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.537813 28669 tablet_server.cc:196] TabletServer@127.27.255.67:0 shutdown complete.
I20260505 20:18:46.557861 28669 master.cc:562] Master@127.27.255.126:40673 shutting down...
I20260505 20:18:46.592069 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:46.592904 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.593286 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2e02ce4bdc754528a079bdb143064843: stopping tablet replica
I20260505 20:18:46.619031 28669 master.cc:584] Master@127.27.255.126:40673 shutdown complete.
W20260505 20:18:46.625696 30727 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 -> Peer 2e02ce4bdc754528a079bdb143064843 (127.27.255.126:40673): Couldn't send request to peer 2e02ce4bdc754528a079bdb143064843. Status: Network error: Client connection negotiation failed: client connection to 127.27.255.126:40673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260505 20:18:46.640906 28669 master.cc:562] Master@127.27.255.125:45869 shutting down...
I20260505 20:18:46.671088 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 6 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:46.671873 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce [term 6 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.672492 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P ef83e6b036a84a31b76b0df6a080dbce: stopping tablet replica
I20260505 20:18:46.698004 28669 master.cc:584] Master@127.27.255.125:45869 shutdown complete.
I20260505 20:18:46.717902 28669 master.cc:562] Master@127.27.255.124:41987 shutting down...
I20260505 20:18:46.739509 30903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "6f405746a59b4a56ad59181504704852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e02ce4bdc754528a079bdb143064843" member_type: VOTER last_known_addr { host: "127.27.255.126" port: 40673 } } peers { permanent_uuid: "ef83e6b036a84a31b76b0df6a080dbce" member_type: VOTER last_known_addr { host: "127.27.255.125" port: 45869 } } peers { permanent_uuid: "6f405746a59b4a56ad59181504704852" member_type: VOTER last_known_addr { host: "127.27.255.124" port: 41987 } } }
I20260505 20:18:46.740587 30903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [sys.catalog]: This master's current role is: LEADER
W20260505 20:18:46.751715 30727 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 -> Peer ef83e6b036a84a31b76b0df6a080dbce (127.27.255.125:45869): Couldn't send request to peer ef83e6b036a84a31b76b0df6a080dbce. Status: Network error: Client connection negotiation failed: client connection to 127.27.255.125:45869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260505 20:18:46.762301 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 6 LEADER]: Raft consensus shutting down.
I20260505 20:18:46.763494 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6f405746a59b4a56ad59181504704852 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:46.764128 28669 tablet_replica.cc:333] stopping tablet replica
I20260505 20:18:46.807299 28669 master.cc:584] Master@127.27.255.124:41987 shutdown complete.
I20260505 20:18:46.834782 28669 test_util.cc:182] -----------------------------------------------
I20260505 20:18:46.834997 28669 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1778012272869005-28669-0
[ FAILED ] TsTabletManagerITest.TestTableStats (37635 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20260505 20:18:46.841840 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:37157
I20260505 20:18:46.843017 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:46.849150 30919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:46.849731 30920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:46.850703 30922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:46.851076 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:46.854278 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:46.854476 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:46.854600 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012326854589 us; error 0 us; skew 500 ppm
I20260505 20:18:46.855052 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:46.858670 28669 webserver.cc:492] Webserver started at http://127.27.255.126:33535/ using document root <none> and password file <none>
I20260505 20:18:46.859193 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:46.859354 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:46.859596 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:46.860775 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "d0b23e3885024f6e85791c883d60eab1"
format_stamp: "Formatted at 2026-05-05 20:18:46 on dist-test-slave-dnnj"
I20260505 20:18:46.865947 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260505 20:18:46.869581 30927 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:46.870573 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260505 20:18:46.870955 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "d0b23e3885024f6e85791c883d60eab1"
format_stamp: "Formatted at 2026-05-05 20:18:46 on dist-test-slave-dnnj"
I20260505 20:18:46.871268 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:46.898378 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:46.899621 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:46.943323 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:37157
I20260505 20:18:46.943434 30978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:37157 every 8 connection(s)
I20260505 20:18:46.948060 30979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:46.960047 30979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1: Bootstrap starting.
I20260505 20:18:46.965164 30979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:46.969439 30979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1: No bootstrap required, opened a new log
I20260505 20:18:46.971758 30979 raft_consensus.cc:359] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b23e3885024f6e85791c883d60eab1" member_type: VOTER }
I20260505 20:18:46.972153 30979 raft_consensus.cc:385] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:46.972420 30979 raft_consensus.cc:740] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b23e3885024f6e85791c883d60eab1, State: Initialized, Role: FOLLOWER
I20260505 20:18:46.973052 30979 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b23e3885024f6e85791c883d60eab1" member_type: VOTER }
I20260505 20:18:46.973614 30979 raft_consensus.cc:399] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:46.973886 30979 raft_consensus.cc:493] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:46.974167 30979 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:46.979192 30979 raft_consensus.cc:515] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b23e3885024f6e85791c883d60eab1" member_type: VOTER }
I20260505 20:18:46.979766 30979 leader_election.cc:304] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0b23e3885024f6e85791c883d60eab1; no voters:
I20260505 20:18:46.980988 30979 leader_election.cc:290] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:46.981333 30982 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:46.982843 30982 raft_consensus.cc:697] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 1 LEADER]: Becoming Leader. State: Replica: d0b23e3885024f6e85791c883d60eab1, State: Running, Role: LEADER
I20260505 20:18:46.983608 30982 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b23e3885024f6e85791c883d60eab1" member_type: VOTER }
I20260505 20:18:46.983879 30979 sys_catalog.cc:565] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:46.986814 30983 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0b23e3885024f6e85791c883d60eab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b23e3885024f6e85791c883d60eab1" member_type: VOTER } }
I20260505 20:18:46.986824 30984 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0b23e3885024f6e85791c883d60eab1. Latest consensus state: current_term: 1 leader_uuid: "d0b23e3885024f6e85791c883d60eab1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b23e3885024f6e85791c883d60eab1" member_type: VOTER } }
I20260505 20:18:46.987838 30983 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:46.987840 30984 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:46.991597 30988 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:46.996742 30988 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:47.001346 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:47.006165 30988 catalog_manager.cc:1357] Generated new cluster ID: e399140a510047b89655599bae6018d3
I20260505 20:18:47.006547 30988 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:47.039496 30988 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:47.040845 30988 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:47.056432 30988 catalog_manager.cc:6044] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1: Generated new TSK 0
I20260505 20:18:47.057040 30988 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:47.068380 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:47.075958 31001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:47.076395 31000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:47.078750 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:47.078503 31003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:47.080035 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:47.080276 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:47.080613 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012327080431 us; error 0 us; skew 500 ppm
I20260505 20:18:47.081760 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:47.084230 28669 webserver.cc:492] Webserver started at http://127.27.255.65:37671/ using document root <none> and password file <none>
I20260505 20:18:47.084756 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:47.084964 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:47.085251 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:47.086494 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "75ca1893938f408d8e7733c1ccf50ced"
format_stamp: "Formatted at 2026-05-05 20:18:47 on dist-test-slave-dnnj"
I20260505 20:18:47.091228 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260505 20:18:47.095199 31008 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:47.096215 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260505 20:18:47.096710 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "75ca1893938f408d8e7733c1ccf50ced"
format_stamp: "Formatted at 2026-05-05 20:18:47 on dist-test-slave-dnnj"
I20260505 20:18:47.097314 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:47.112160 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:47.113343 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:47.115363 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:47.118013 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:47.118209 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:47.118502 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:47.118717 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:47.165570 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:36905
I20260505 20:18:47.165763 31070 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:36905 every 8 connection(s)
I20260505 20:18:47.170281 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:47.178591 31075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:47.180706 31076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:47.184345 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:47.185017 31078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:47.186194 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:47.186475 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:47.186702 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012327186684 us; error 0 us; skew 500 ppm
I20260505 20:18:47.187273 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:47.191668 28669 webserver.cc:492] Webserver started at http://127.27.255.66:37383/ using document root <none> and password file <none>
I20260505 20:18:47.192384 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:47.192636 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:47.192915 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:47.193640 31071 heartbeater.cc:344] Connected to a master server at 127.27.255.126:37157
I20260505 20:18:47.194407 31071 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:47.194869 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-1-root/instance:
uuid: "81befde1125a419db22ea0f924ec4d62"
format_stamp: "Formatted at 2026-05-05 20:18:47 on dist-test-slave-dnnj"
I20260505 20:18:47.195420 31071 heartbeater.cc:507] Master 127.27.255.126:37157 requested a full tablet report, sending...
I20260505 20:18:47.197890 30944 ts_manager.cc:194] Registered new tserver with Master: 75ca1893938f408d8e7733c1ccf50ced (127.27.255.65:36905)
I20260505 20:18:47.199851 30944 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44432
I20260505 20:18:47.203181 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260505 20:18:47.207274 31083 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:47.208417 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260505 20:18:47.208746 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-1-root
uuid: "81befde1125a419db22ea0f924ec4d62"
format_stamp: "Formatted at 2026-05-05 20:18:47 on dist-test-slave-dnnj"
I20260505 20:18:47.209012 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:47.234241 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:47.235553 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:47.237041 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:47.239316 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:47.239504 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:47.239781 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:47.239925 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:47.284250 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.66:35379
I20260505 20:18:47.284351 31145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.66:35379 every 8 connection(s)
I20260505 20:18:47.289696 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:47.297472 31149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:47.300529 31150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:47.301142 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:47.302053 31152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:47.303158 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:47.303570 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:47.303700 31146 heartbeater.cc:344] Connected to a master server at 127.27.255.126:37157
I20260505 20:18:47.303799 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012327303780 us; error 0 us; skew 500 ppm
I20260505 20:18:47.304262 31146 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:47.304684 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:47.305027 31146 heartbeater.cc:507] Master 127.27.255.126:37157 requested a full tablet report, sending...
I20260505 20:18:47.307514 28669 webserver.cc:492] Webserver started at http://127.27.255.67:46597/ using document root <none> and password file <none>
I20260505 20:18:47.307504 30944 ts_manager.cc:194] Registered new tserver with Master: 81befde1125a419db22ea0f924ec4d62 (127.27.255.66:35379)
I20260505 20:18:47.308382 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:47.308668 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:47.308979 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:47.309545 30944 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44448
I20260505 20:18:47.310624 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-2-root/instance:
uuid: "7c52397e58a240278c0f82ae7b73c1bc"
format_stamp: "Formatted at 2026-05-05 20:18:47 on dist-test-slave-dnnj"
I20260505 20:18:47.315558 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260505 20:18:47.319006 31157 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:47.319914 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260505 20:18:47.320243 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-2-root
uuid: "7c52397e58a240278c0f82ae7b73c1bc"
format_stamp: "Formatted at 2026-05-05 20:18:47 on dist-test-slave-dnnj"
I20260505 20:18:47.320565 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:47.342223 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:47.343456 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:47.344934 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:47.352391 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:47.352702 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:47.353223 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:47.353588 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.002s sys 0.000s
I20260505 20:18:47.401990 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.67:35699
I20260505 20:18:47.402096 31219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.67:35699 every 8 connection(s)
I20260505 20:18:47.416478 31220 heartbeater.cc:344] Connected to a master server at 127.27.255.126:37157
I20260505 20:18:47.416828 31220 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:47.417522 31220 heartbeater.cc:507] Master 127.27.255.126:37157 requested a full tablet report, sending...
I20260505 20:18:47.419452 30944 ts_manager.cc:194] Registered new tserver with Master: 7c52397e58a240278c0f82ae7b73c1bc (127.27.255.67:35699)
I20260505 20:18:47.420269 28669 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01504336s
I20260505 20:18:47.421321 30944 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44450
I20260505 20:18:47.460469 30944 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260505 20:18:47.462612 30944 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260505 20:18:47.506711 31185 tablet_service.cc:1511] Processing CreateTablet for tablet 49ecff9addd44e6eb745cf563b08847a (DEFAULT_TABLE table=test-table [id=4bc66eb7e1c44300b9128e61fd79f05d]), partition=
I20260505 20:18:47.507933 31185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49ecff9addd44e6eb745cf563b08847a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:47.509624 31036 tablet_service.cc:1511] Processing CreateTablet for tablet 49ecff9addd44e6eb745cf563b08847a (DEFAULT_TABLE table=test-table [id=4bc66eb7e1c44300b9128e61fd79f05d]), partition=
I20260505 20:18:47.510838 31036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49ecff9addd44e6eb745cf563b08847a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:47.522826 31111 tablet_service.cc:1511] Processing CreateTablet for tablet 49ecff9addd44e6eb745cf563b08847a (DEFAULT_TABLE table=test-table [id=4bc66eb7e1c44300b9128e61fd79f05d]), partition=
I20260505 20:18:47.524047 31111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49ecff9addd44e6eb745cf563b08847a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:47.523988 31236 tablet_bootstrap.cc:492] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: Bootstrap starting.
I20260505 20:18:47.530476 31236 tablet_bootstrap.cc:654] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:47.531301 31237 tablet_bootstrap.cc:492] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: Bootstrap starting.
I20260505 20:18:47.536408 31236 tablet_bootstrap.cc:492] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: No bootstrap required, opened a new log
I20260505 20:18:47.537029 31236 ts_tablet_manager.cc:1403] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.009s
I20260505 20:18:47.538983 31237 tablet_bootstrap.cc:654] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:47.540386 31236 raft_consensus.cc:359] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.541280 31236 raft_consensus.cc:385] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:47.541728 31239 tablet_bootstrap.cc:492] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Bootstrap starting.
I20260505 20:18:47.541728 31236 raft_consensus.cc:740] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c52397e58a240278c0f82ae7b73c1bc, State: Initialized, Role: FOLLOWER
I20260505 20:18:47.542964 31236 consensus_queue.cc:260] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.544798 31237 tablet_bootstrap.cc:492] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: No bootstrap required, opened a new log
I20260505 20:18:47.545322 31237 ts_tablet_manager.cc:1403] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20260505 20:18:47.545822 31220 heartbeater.cc:499] Master 127.27.255.126:37157 was elected leader, sending a full tablet report...
I20260505 20:18:47.547174 31236 ts_tablet_manager.cc:1434] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260505 20:18:47.547688 31236 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260505 20:18:47.548899 31237 raft_consensus.cc:359] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.549782 31237 raft_consensus.cc:385] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:47.550030 31237 raft_consensus.cc:740] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ca1893938f408d8e7733c1ccf50ced, State: Initialized, Role: FOLLOWER
I20260505 20:18:47.550534 31239 tablet_bootstrap.cc:654] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:47.550668 31237 consensus_queue.cc:260] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.552974 31071 heartbeater.cc:499] Master 127.27.255.126:37157 was elected leader, sending a full tablet report...
I20260505 20:18:47.555356 31237 ts_tablet_manager.cc:1434] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
W20260505 20:18:47.555631 31237 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260505 20:18:47.558081 31239 tablet_bootstrap.cc:492] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: No bootstrap required, opened a new log
I20260505 20:18:47.558619 31239 ts_tablet_manager.cc:1403] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260505 20:18:47.561444 31239 raft_consensus.cc:359] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.562096 31239 raft_consensus.cc:385] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:47.562310 31239 raft_consensus.cc:740] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81befde1125a419db22ea0f924ec4d62, State: Initialized, Role: FOLLOWER
I20260505 20:18:47.562999 31239 consensus_queue.cc:260] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.564076 31242 raft_consensus.cc:493] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:47.564911 31242 raft_consensus.cc:515] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.565485 31146 heartbeater.cc:499] Master 127.27.255.126:37157 was elected leader, sending a full tablet report...
I20260505 20:18:47.566032 31239 ts_tablet_manager.cc:1434] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
W20260505 20:18:47.566303 31239 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260505 20:18:47.568265 31242 leader_election.cc:290] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c52397e58a240278c0f82ae7b73c1bc (127.27.255.67:35699), 81befde1125a419db22ea0f924ec4d62 (127.27.255.66:35379)
I20260505 20:18:47.577677 31195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ecff9addd44e6eb745cf563b08847a" candidate_uuid: "75ca1893938f408d8e7733c1ccf50ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c52397e58a240278c0f82ae7b73c1bc" is_pre_election: true
I20260505 20:18:47.578439 31195 raft_consensus.cc:2468] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75ca1893938f408d8e7733c1ccf50ced in term 0.
I20260505 20:18:47.579759 31010 leader_election.cc:304] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75ca1893938f408d8e7733c1ccf50ced, 7c52397e58a240278c0f82ae7b73c1bc; no voters:
I20260505 20:18:47.580686 31242 raft_consensus.cc:2804] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260505 20:18:47.580840 31121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ecff9addd44e6eb745cf563b08847a" candidate_uuid: "75ca1893938f408d8e7733c1ccf50ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81befde1125a419db22ea0f924ec4d62" is_pre_election: true
I20260505 20:18:47.581103 31242 raft_consensus.cc:493] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:47.581611 31121 raft_consensus.cc:2468] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75ca1893938f408d8e7733c1ccf50ced in term 0.
I20260505 20:18:47.581785 31242 raft_consensus.cc:3060] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:47.588515 31242 raft_consensus.cc:515] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.590830 31242 leader_election.cc:290] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [CANDIDATE]: Term 1 election: Requested vote from peers 7c52397e58a240278c0f82ae7b73c1bc (127.27.255.67:35699), 81befde1125a419db22ea0f924ec4d62 (127.27.255.66:35379)
I20260505 20:18:47.591655 31195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ecff9addd44e6eb745cf563b08847a" candidate_uuid: "75ca1893938f408d8e7733c1ccf50ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c52397e58a240278c0f82ae7b73c1bc"
I20260505 20:18:47.591907 31121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49ecff9addd44e6eb745cf563b08847a" candidate_uuid: "75ca1893938f408d8e7733c1ccf50ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81befde1125a419db22ea0f924ec4d62"
I20260505 20:18:47.592197 31195 raft_consensus.cc:3060] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:47.592592 31121 raft_consensus.cc:3060] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:47.598145 31195 raft_consensus.cc:2468] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75ca1893938f408d8e7733c1ccf50ced in term 1.
I20260505 20:18:47.598142 31121 raft_consensus.cc:2468] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75ca1893938f408d8e7733c1ccf50ced in term 1.
I20260505 20:18:47.599611 31010 leader_election.cc:304] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75ca1893938f408d8e7733c1ccf50ced, 7c52397e58a240278c0f82ae7b73c1bc; no voters:
I20260505 20:18:47.600432 31242 raft_consensus.cc:2804] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:47.601609 31242 raft_consensus.cc:697] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 1 LEADER]: Becoming Leader. State: Replica: 75ca1893938f408d8e7733c1ccf50ced, State: Running, Role: LEADER
I20260505 20:18:47.602249 31242 consensus_queue.cc:237] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.610493 30943 catalog_manager.cc:5671] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced reported cstate change: term changed from 0 to 1, leader changed from <none> to 75ca1893938f408d8e7733c1ccf50ced (127.27.255.65). New cstate: current_term: 1 leader_uuid: "75ca1893938f408d8e7733c1ccf50ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } health_report { overall_health: UNKNOWN } } }
I20260505 20:18:47.646546 31111 tablet_service.cc:1558] Processing DeleteTablet for tablet 49ecff9addd44e6eb745cf563b08847a with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:59258
I20260505 20:18:47.648478 31249 tablet_replica.cc:333] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: stopping tablet replica
I20260505 20:18:47.649024 31249 raft_consensus.cc:2243] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:47.649412 31249 raft_consensus.cc:2272] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:47.652388 31249 ts_tablet_manager.cc:1916] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260505 20:18:47.656067 31221 tablet.cc:2404] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260505 20:18:47.664793 31249 ts_tablet_manager.cc:1929] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20260505 20:18:47.665181 31249 log.cc:1199] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Deleting WAL directory at /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-1-root/wals/49ecff9addd44e6eb745cf563b08847a
I20260505 20:18:47.670207 31250 ts_tablet_manager.cc:933] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Initiating tablet copy from peer 75ca1893938f408d8e7733c1ccf50ced (127.27.255.65:36905)
I20260505 20:18:47.672223 31250 tablet_copy_client.cc:323] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: tablet copy: Beginning tablet copy session from remote peer at address 127.27.255.65:36905
I20260505 20:18:47.684783 31056 tablet_copy_service.cc:140] P 75ca1893938f408d8e7733c1ccf50ced: Received BeginTabletCopySession request for tablet 49ecff9addd44e6eb745cf563b08847a from peer 81befde1125a419db22ea0f924ec4d62 ({username='slave'} at 127.0.0.1:60946)
I20260505 20:18:47.685483 31056 tablet_copy_service.cc:161] P 75ca1893938f408d8e7733c1ccf50ced: Beginning new tablet copy session on tablet 49ecff9addd44e6eb745cf563b08847a from peer 81befde1125a419db22ea0f924ec4d62 at {username='slave'} at 127.0.0.1:60946: session id = 81befde1125a419db22ea0f924ec4d62-49ecff9addd44e6eb745cf563b08847a
I20260505 20:18:47.693023 31056 tablet_copy_source_session.cc:215] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: Tablet Copy: opened 0 blocks and 1 log segments
I20260505 20:18:47.696441 31250 ts_tablet_manager.cc:1916] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260505 20:18:47.707366 31250 ts_tablet_manager.cc:1929] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20260505 20:18:47.708045 31250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49ecff9addd44e6eb745cf563b08847a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:47.713781 31250 tablet_copy_client.cc:806] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: tablet copy: Starting download of 0 data blocks...
I20260505 20:18:47.714334 31250 tablet_copy_client.cc:670] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: tablet copy: Starting download of 1 WAL segments...
I20260505 20:18:47.716609 30943 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44466:
table { table_id: "4bc66eb7e1c44300b9128e61fd79f05d" table_name: "test-table" }
I20260505 20:18:47.717283 30943 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44466:
table { table_id: "4bc66eb7e1c44300b9128e61fd79f05d" table_name: "test-table" }
I20260505 20:18:47.718536 31250 tablet_copy_client.cc:538] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260505 20:18:47.726644 31250 tablet_bootstrap.cc:492] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Bootstrap starting.
I20260505 20:18:47.729131 30943 catalog_manager.cc:5958] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1: Sending DeleteTablet for 3 replicas of tablet 49ecff9addd44e6eb745cf563b08847a
I20260505 20:18:47.731441 31185 tablet_service.cc:1558] Processing DeleteTablet for tablet 49ecff9addd44e6eb745cf563b08847a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-05 20:18:47 UTC) from {username='slave'} at 127.0.0.1:38938
I20260505 20:18:47.731617 31036 tablet_service.cc:1558] Processing DeleteTablet for tablet 49ecff9addd44e6eb745cf563b08847a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-05 20:18:47 UTC) from {username='slave'} at 127.0.0.1:60940
I20260505 20:18:47.732792 31111 tablet_service.cc:1558] Processing DeleteTablet for tablet 49ecff9addd44e6eb745cf563b08847a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-05 20:18:47 UTC) from {username='slave'} at 127.0.0.1:59234
W20260505 20:18:47.735219 30929 catalog_manager.cc:4982] TS 81befde1125a419db22ea0f924ec4d62 (127.27.255.66:35379): delete failed for tablet 49ecff9addd44e6eb745cf563b08847a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49ecff9addd44e6eb745cf563b08847a already in progress: copying tablet
I20260505 20:18:47.734576 31253 tablet_replica.cc:333] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: stopping tablet replica
I20260505 20:18:47.736745 31252 tablet_replica.cc:333] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: stopping tablet replica
I20260505 20:18:47.736680 31253 raft_consensus.cc:2243] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:47.737638 31252 raft_consensus.cc:2243] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:47.737763 31253 pending_rounds.cc:70] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: Trying to abort 1 pending ops.
I20260505 20:18:47.738303 31252 raft_consensus.cc:2272] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:47.738209 31253 pending_rounds.cc:77] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282738493865381888 op_type: NO_OP noop_request { }
I20260505 20:18:47.738770 31253 raft_consensus.cc:2889] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260505 20:18:47.739296 31253 raft_consensus.cc:2272] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:47.741541 31252 ts_tablet_manager.cc:1916] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260505 20:18:47.743229 31253 ts_tablet_manager.cc:1916] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260505 20:18:47.744886 31250 tablet_bootstrap.cc:492] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260505 20:18:47.746234 31250 tablet_bootstrap.cc:492] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Bootstrap complete.
I20260505 20:18:47.747005 31250 ts_tablet_manager.cc:1403] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.012s
I20260505 20:18:47.750100 31250 raft_consensus.cc:359] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.751042 31250 raft_consensus.cc:740] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81befde1125a419db22ea0f924ec4d62, State: Initialized, Role: FOLLOWER
I20260505 20:18:47.751940 31250 consensus_queue.cc:260] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c52397e58a240278c0f82ae7b73c1bc" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 35699 } } peers { permanent_uuid: "75ca1893938f408d8e7733c1ccf50ced" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36905 } } peers { permanent_uuid: "81befde1125a419db22ea0f924ec4d62" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 35379 } }
I20260505 20:18:47.754045 31250 ts_tablet_manager.cc:1434] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260505 20:18:47.754307 31250 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260505 20:18:47.755666 31253 ts_tablet_manager.cc:1929] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260505 20:18:47.756163 31253 log.cc:1199] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: Deleting WAL directory at /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-0-root/wals/49ecff9addd44e6eb745cf563b08847a
I20260505 20:18:47.756902 31253 ts_tablet_manager.cc:1950] T 49ecff9addd44e6eb745cf563b08847a P 75ca1893938f408d8e7733c1ccf50ced: Deleting consensus metadata
I20260505 20:18:47.757226 31252 ts_tablet_manager.cc:1929] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260505 20:18:47.757675 31252 log.cc:1199] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: Deleting WAL directory at /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-2-root/wals/49ecff9addd44e6eb745cf563b08847a
I20260505 20:18:47.758525 31252 ts_tablet_manager.cc:1950] T 49ecff9addd44e6eb745cf563b08847a P 7c52397e58a240278c0f82ae7b73c1bc: Deleting consensus metadata
I20260505 20:18:47.758500 30931 catalog_manager.cc:5002] TS 75ca1893938f408d8e7733c1ccf50ced (127.27.255.65:36905): tablet 49ecff9addd44e6eb745cf563b08847a (table test-table [id=4bc66eb7e1c44300b9128e61fd79f05d]) successfully deleted
I20260505 20:18:47.759920 31111 tablet_service.cc:1558] Processing DeleteTablet for tablet 49ecff9addd44e6eb745cf563b08847a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-05 20:18:47 UTC) from {username='slave'} at 127.0.0.1:59234
I20260505 20:18:47.761348 30929 catalog_manager.cc:5002] TS 7c52397e58a240278c0f82ae7b73c1bc (127.27.255.67:35699): tablet 49ecff9addd44e6eb745cf563b08847a (table test-table [id=4bc66eb7e1c44300b9128e61fd79f05d]) successfully deleted
W20260505 20:18:47.762560 30929 catalog_manager.cc:4982] TS 81befde1125a419db22ea0f924ec4d62 (127.27.255.66:35379): delete failed for tablet 49ecff9addd44e6eb745cf563b08847a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49ecff9addd44e6eb745cf563b08847a already in progress: copying tablet
I20260505 20:18:47.807031 31056 tablet_copy_service.cc:342] P 75ca1893938f408d8e7733c1ccf50ced: Request end of tablet copy session 81befde1125a419db22ea0f924ec4d62-49ecff9addd44e6eb745cf563b08847a received from {username='slave'} at 127.0.0.1:60946
I20260505 20:18:47.807617 31056 tablet_copy_service.cc:434] P 75ca1893938f408d8e7733c1ccf50ced: ending tablet copy session 81befde1125a419db22ea0f924ec4d62-49ecff9addd44e6eb745cf563b08847a on tablet 49ecff9addd44e6eb745cf563b08847a with peer 81befde1125a419db22ea0f924ec4d62
I20260505 20:18:47.813187 31111 tablet_service.cc:1558] Processing DeleteTablet for tablet 49ecff9addd44e6eb745cf563b08847a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-05 20:18:47 UTC) from {username='slave'} at 127.0.0.1:59234
I20260505 20:18:47.814131 31249 tablet_replica.cc:333] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: stopping tablet replica
I20260505 20:18:47.814987 31249 raft_consensus.cc:2243] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:47.815481 31249 pending_rounds.cc:70] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Trying to abort 1 pending ops.
I20260505 20:18:47.815726 31249 pending_rounds.cc:77] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7282738493865381888 op_type: NO_OP noop_request { }
I20260505 20:18:47.816058 31249 raft_consensus.cc:2889] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260505 20:18:47.816421 31249 raft_consensus.cc:2272] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:47.819219 31249 ts_tablet_manager.cc:1916] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260505 20:18:47.832569 31249 ts_tablet_manager.cc:1929] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260505 20:18:47.833007 31249 log.cc:1199] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Deleting WAL directory at /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1778012272869005-28669-0/minicluster-data/ts-1-root/wals/49ecff9addd44e6eb745cf563b08847a
I20260505 20:18:47.833787 31249 ts_tablet_manager.cc:1950] T 49ecff9addd44e6eb745cf563b08847a P 81befde1125a419db22ea0f924ec4d62: Deleting consensus metadata
I20260505 20:18:47.836516 30929 catalog_manager.cc:5002] TS 81befde1125a419db22ea0f924ec4d62 (127.27.255.66:35379): tablet 49ecff9addd44e6eb745cf563b08847a (table test-table [id=4bc66eb7e1c44300b9128e61fd79f05d]) successfully deleted
I20260505 20:18:47.870664 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:47.898175 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:47.917618 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:47.929620 28669 tablet_server.cc:179] TabletServer@127.27.255.66:0 shutting down...
I20260505 20:18:47.952117 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:47.972013 28669 tablet_server.cc:196] TabletServer@127.27.255.66:0 shutdown complete.
I20260505 20:18:47.984577 28669 tablet_server.cc:179] TabletServer@127.27.255.67:0 shutting down...
I20260505 20:18:48.004271 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:48.023574 28669 tablet_server.cc:196] TabletServer@127.27.255.67:0 shutdown complete.
I20260505 20:18:48.035203 28669 master.cc:562] Master@127.27.255.126:37157 shutting down...
I20260505 20:18:48.052950 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:48.053617 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:48.053932 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P d0b23e3885024f6e85791c883d60eab1: stopping tablet replica
I20260505 20:18:48.065109 28669 master.cc:584] Master@127.27.255.126:37157 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1252 ms)
[----------] 4 tests from TsTabletManagerITest (43996 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20260505 20:18:48.095551 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:34721
I20260505 20:18:48.096781 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:48.102783 31259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:48.103834 31260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:48.105778 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:48.106062 31262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:48.107025 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:48.107213 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:48.107400 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012328107383 us; error 0 us; skew 500 ppm
I20260505 20:18:48.108139 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:48.111126 28669 webserver.cc:492] Webserver started at http://127.27.255.126:43677/ using document root <none> and password file <none>
I20260505 20:18:48.111692 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:48.111896 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:48.112200 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:48.113477 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "a54366bef653423aac59ea668d410043"
format_stamp: "Formatted at 2026-05-05 20:18:48 on dist-test-slave-dnnj"
I20260505 20:18:48.118510 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260505 20:18:48.122738 31267 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:48.123788 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260505 20:18:48.124114 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "a54366bef653423aac59ea668d410043"
format_stamp: "Formatted at 2026-05-05 20:18:48 on dist-test-slave-dnnj"
I20260505 20:18:48.124406 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:48.138760 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:48.139989 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:48.186013 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:34721
I20260505 20:18:48.186112 31318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:34721 every 8 connection(s)
I20260505 20:18:48.191025 31319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:48.203687 31319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043: Bootstrap starting.
I20260505 20:18:48.209060 31319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:48.213774 31319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043: No bootstrap required, opened a new log
I20260505 20:18:48.216089 31319 raft_consensus.cc:359] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54366bef653423aac59ea668d410043" member_type: VOTER }
I20260505 20:18:48.216542 31319 raft_consensus.cc:385] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:48.216730 31319 raft_consensus.cc:740] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a54366bef653423aac59ea668d410043, State: Initialized, Role: FOLLOWER
I20260505 20:18:48.217361 31319 consensus_queue.cc:260] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54366bef653423aac59ea668d410043" member_type: VOTER }
I20260505 20:18:48.217859 31319 raft_consensus.cc:399] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:48.218055 31319 raft_consensus.cc:493] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:48.218268 31319 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:48.223623 31319 raft_consensus.cc:515] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54366bef653423aac59ea668d410043" member_type: VOTER }
I20260505 20:18:48.224195 31319 leader_election.cc:304] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a54366bef653423aac59ea668d410043; no voters:
I20260505 20:18:48.225512 31319 leader_election.cc:290] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:48.225934 31322 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:48.227679 31322 raft_consensus.cc:697] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 1 LEADER]: Becoming Leader. State: Replica: a54366bef653423aac59ea668d410043, State: Running, Role: LEADER
I20260505 20:18:48.228399 31322 consensus_queue.cc:237] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54366bef653423aac59ea668d410043" member_type: VOTER }
I20260505 20:18:48.229142 31319 sys_catalog.cc:565] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:48.233814 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a54366bef653423aac59ea668d410043. Latest consensus state: current_term: 1 leader_uuid: "a54366bef653423aac59ea668d410043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54366bef653423aac59ea668d410043" member_type: VOTER } }
I20260505 20:18:48.234556 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:48.234537 31323 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a54366bef653423aac59ea668d410043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54366bef653423aac59ea668d410043" member_type: VOTER } }
I20260505 20:18:48.235513 31323 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:48.238471 31330 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:48.243710 31330 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:48.247330 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:48.253795 31330 catalog_manager.cc:1357] Generated new cluster ID: 8256af0dc3bc4a27922b60d1dd8ced41
I20260505 20:18:48.254148 31330 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:48.276877 31330 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:48.279093 31330 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:48.296207 31330 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043: Generated new TSK 0
I20260505 20:18:48.296983 31330 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:48.315407 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:48.322423 31340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:48.323361 31341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:48.325222 31343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:48.326356 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:48.327292 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:48.327543 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:48.327801 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012328327699 us; error 0 us; skew 500 ppm
I20260505 20:18:48.328375 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:48.331519 28669 webserver.cc:492] Webserver started at http://127.27.255.65:40419/ using document root <none> and password file <none>
I20260505 20:18:48.332091 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:48.332324 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:48.332623 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:48.334144 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "5f45bd546fe64878a92c1b1b6f659abc"
format_stamp: "Formatted at 2026-05-05 20:18:48 on dist-test-slave-dnnj"
I20260505 20:18:48.339349 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260505 20:18:48.343658 31348 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:48.344604 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:48.345024 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "5f45bd546fe64878a92c1b1b6f659abc"
format_stamp: "Formatted at 2026-05-05 20:18:48 on dist-test-slave-dnnj"
I20260505 20:18:48.345397 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:48.361049 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:48.362469 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:48.368167 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:48.368636 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260505 20:18:48.369048 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:48.369331 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:48.446106 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:45687
I20260505 20:18:48.446231 31418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:45687 every 8 connection(s)
I20260505 20:18:48.452179 31419 heartbeater.cc:344] Connected to a master server at 127.27.255.126:34721
I20260505 20:18:48.452754 31419 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:48.453737 31419 heartbeater.cc:507] Master 127.27.255.126:34721 requested a full tablet report, sending...
I20260505 20:18:48.456403 31284 ts_manager.cc:194] Registered new tserver with Master: 5f45bd546fe64878a92c1b1b6f659abc (127.27.255.65:45687)
I20260505 20:18:48.457289 28669 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.006824156s
I20260505 20:18:48.458702 31284 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46460
I20260505 20:18:48.499605 31384 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260505 20:18:48.501236 31384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:48.516242 31429 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: Bootstrap starting.
I20260505 20:18:48.522128 31429 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:48.527695 31429 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: No bootstrap required, opened a new log
I20260505 20:18:48.528249 31429 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260505 20:18:48.530784 31429 raft_consensus.cc:359] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } }
I20260505 20:18:48.531337 31429 raft_consensus.cc:385] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:48.531589 31429 raft_consensus.cc:740] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f45bd546fe64878a92c1b1b6f659abc, State: Initialized, Role: FOLLOWER
I20260505 20:18:48.532248 31429 consensus_queue.cc:260] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } }
I20260505 20:18:48.532766 31429 raft_consensus.cc:399] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:48.533020 31429 raft_consensus.cc:493] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:48.533291 31429 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:48.539331 31429 raft_consensus.cc:515] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } }
I20260505 20:18:48.540071 31429 leader_election.cc:304] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f45bd546fe64878a92c1b1b6f659abc; no voters:
I20260505 20:18:48.541733 31429 leader_election.cc:290] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:48.542145 31431 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:48.543898 31431 raft_consensus.cc:697] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 1 LEADER]: Becoming Leader. State: Replica: 5f45bd546fe64878a92c1b1b6f659abc, State: Running, Role: LEADER
I20260505 20:18:48.543917 31419 heartbeater.cc:499] Master 127.27.255.126:34721 was elected leader, sending a full tablet report...
I20260505 20:18:48.544924 31429 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: Time spent starting tablet: real 0.016s user 0.008s sys 0.009s
I20260505 20:18:48.544867 31431 consensus_queue.cc:237] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } }
I20260505 20:18:48.550733 31433 tablet_replica.cc:442] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: TxnStatusTablet state changed. Reason: New leader 5f45bd546fe64878a92c1b1b6f659abc. Latest consensus state: current_term: 1 leader_uuid: "5f45bd546fe64878a92c1b1b6f659abc" committed_config { opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } } }
I20260505 20:18:48.550438 31432 tablet_replica.cc:442] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f45bd546fe64878a92c1b1b6f659abc" committed_config { opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } } }
I20260505 20:18:48.551394 31433 tablet_replica.cc:445] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: This TxnStatusTablet replica's current role is: LEADER
I20260505 20:18:48.551542 31432 tablet_replica.cc:445] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: This TxnStatusTablet replica's current role is: LEADER
W20260505 20:18:48.553802 31284 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:48.554184 31436 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260505 20:18:48.554683 31436 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260505 20:18:48.559993 31284 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:48.596626 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:48.628254 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:48.629271 28669 tablet_replica.cc:333] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: stopping tablet replica
I20260505 20:18:48.629966 28669 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:48.630656 28669 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:48.656071 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:48.670642 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:48.677896 31440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:48.677997 31439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:48.682224 31442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:48.683034 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:48.684504 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:48.684811 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:48.685015 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012328684997 us; error 0 us; skew 500 ppm
I20260505 20:18:48.685887 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:48.689265 28669 webserver.cc:492] Webserver started at http://127.27.255.65:40419/ using document root <none> and password file <none>
I20260505 20:18:48.690034 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:48.690274 28669 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260505 20:18:48.695858 28669 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260505 20:18:48.700304 31447 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:48.701565 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:48.701939 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "5f45bd546fe64878a92c1b1b6f659abc"
format_stamp: "Formatted at 2026-05-05 20:18:48 on dist-test-slave-dnnj"
I20260505 20:18:48.702284 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:48.724519 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:48.725968 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:48.733767 31456 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20260505 20:18:48.744191 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260505 20:18:48.744596 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20260505 20:18:48.744989 28669 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260505 20:18:48.753600 28669 ts_tablet_manager.cc:616] Registered 1 tablets
I20260505 20:18:48.753944 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.009s user 0.009s sys 0.000s
I20260505 20:18:48.754621 31456 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: Bootstrap starting.
I20260505 20:18:48.805097 31456 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 20:18:48.807657 31456 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: Bootstrap complete.
I20260505 20:18:48.808786 31456 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: Time spent bootstrapping tablet: real 0.054s user 0.030s sys 0.018s
I20260505 20:18:48.812057 31456 raft_consensus.cc:359] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } }
I20260505 20:18:48.812829 31456 raft_consensus.cc:740] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f45bd546fe64878a92c1b1b6f659abc, State: Initialized, Role: FOLLOWER
I20260505 20:18:48.813822 31456 consensus_queue.cc:260] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } }
I20260505 20:18:48.814699 31456 raft_consensus.cc:399] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:48.815253 31456 raft_consensus.cc:493] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:48.815653 31456 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:48.824461 31456 raft_consensus.cc:515] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } }
I20260505 20:18:48.825344 31456 leader_election.cc:304] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f45bd546fe64878a92c1b1b6f659abc; no voters:
I20260505 20:18:48.829061 31456 leader_election.cc:290] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260505 20:18:48.829401 31494 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 2 FOLLOWER]: Leader election won for term 2
I20260505 20:18:48.833784 31494 raft_consensus.cc:697] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 2 LEADER]: Becoming Leader. State: Replica: 5f45bd546fe64878a92c1b1b6f659abc, State: Running, Role: LEADER
I20260505 20:18:48.835078 31494 consensus_queue.cc:237] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } }
I20260505 20:18:48.841014 31456 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc: Time spent starting tablet: real 0.032s user 0.025s sys 0.009s
I20260505 20:18:48.853819 31497 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5f45bd546fe64878a92c1b1b6f659abc" committed_config { opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } } }
I20260505 20:18:48.853658 31496 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 5f45bd546fe64878a92c1b1b6f659abc. Latest consensus state: current_term: 2 leader_uuid: "5f45bd546fe64878a92c1b1b6f659abc" committed_config { opid_index: -1 peers { permanent_uuid: "5f45bd546fe64878a92c1b1b6f659abc" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 45687 } } }
I20260505 20:18:48.854530 31497 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260505 20:18:48.854656 31496 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260505 20:18:48.856693 31504 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260505 20:18:48.857168 31504 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260505 20:18:48.881181 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:45687
I20260505 20:18:48.881327 31525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:45687 every 8 connection(s)
I20260505 20:18:48.887264 31526 heartbeater.cc:344] Connected to a master server at 127.27.255.126:34721
I20260505 20:18:48.888005 31526 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:48.889137 31526 heartbeater.cc:507] Master 127.27.255.126:34721 requested a full tablet report, sending...
I20260505 20:18:48.889715 28669 tablet_server.cc:179] TabletServer@127.27.255.65:45687 shutting down...
I20260505 20:18:48.895584 31284 ts_manager.cc:194] Re-registered known tserver with Master: 5f45bd546fe64878a92c1b1b6f659abc (127.27.255.65:45687)
W20260505 20:18:48.896482 31284 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:48.898197 31284 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46500
I20260505 20:18:48.901816 31526 heartbeater.cc:499] Master 127.27.255.126:34721 was elected leader, sending a full tablet report...
W20260505 20:18:48.904696 31284 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:48.921089 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:48.922053 28669 tablet_replica.cc:333] stopping tablet replica
I20260505 20:18:48.922879 28669 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 2 LEADER]: Raft consensus shutting down.
I20260505 20:18:48.923441 28669 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 5f45bd546fe64878a92c1b1b6f659abc [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:48.946187 28669 tablet_server.cc:196] TabletServer@127.27.255.65:45687 shutdown complete.
I20260505 20:18:48.959869 28669 master.cc:562] Master@127.27.255.126:34721 shutting down...
I20260505 20:18:49.001413 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:49.002106 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:49.002489 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P a54366bef653423aac59ea668d410043: stopping tablet replica
I20260505 20:18:49.015268 28669 master.cc:584] Master@127.27.255.126:34721 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (954 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20260505 20:18:49.049408 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:35053
I20260505 20:18:49.050665 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:49.056355 31532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:49.056648 31533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:49.060141 31535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:49.060887 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:49.061844 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:49.062069 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:49.062245 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012329062233 us; error 0 us; skew 500 ppm
I20260505 20:18:49.062827 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:49.065771 28669 webserver.cc:492] Webserver started at http://127.27.255.126:44685/ using document root <none> and password file <none>
I20260505 20:18:49.066270 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:49.066475 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:49.066735 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:49.067806 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "73b47622b2d240ce9200cb39d3ef5d0c"
format_stamp: "Formatted at 2026-05-05 20:18:49 on dist-test-slave-dnnj"
I20260505 20:18:49.072932 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260505 20:18:49.076678 31540 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:49.077714 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260505 20:18:49.078065 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "73b47622b2d240ce9200cb39d3ef5d0c"
format_stamp: "Formatted at 2026-05-05 20:18:49 on dist-test-slave-dnnj"
I20260505 20:18:49.078428 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:49.091020 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:49.092972 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:49.141405 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:35053
I20260505 20:18:49.141855 31591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:35053 every 8 connection(s)
I20260505 20:18:49.146615 31592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:49.157454 31592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c: Bootstrap starting.
I20260505 20:18:49.162492 31592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:49.166867 31592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c: No bootstrap required, opened a new log
I20260505 20:18:49.169163 31592 raft_consensus.cc:359] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b47622b2d240ce9200cb39d3ef5d0c" member_type: VOTER }
I20260505 20:18:49.169838 31592 raft_consensus.cc:385] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:49.170092 31592 raft_consensus.cc:740] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b47622b2d240ce9200cb39d3ef5d0c, State: Initialized, Role: FOLLOWER
I20260505 20:18:49.170763 31592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b47622b2d240ce9200cb39d3ef5d0c" member_type: VOTER }
I20260505 20:18:49.171238 31592 raft_consensus.cc:399] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:49.171496 31592 raft_consensus.cc:493] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:49.171826 31592 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:49.176443 31592 raft_consensus.cc:515] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b47622b2d240ce9200cb39d3ef5d0c" member_type: VOTER }
I20260505 20:18:49.176986 31592 leader_election.cc:304] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73b47622b2d240ce9200cb39d3ef5d0c; no voters:
I20260505 20:18:49.178437 31592 leader_election.cc:290] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:49.178821 31595 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:49.180631 31595 raft_consensus.cc:697] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 1 LEADER]: Becoming Leader. State: Replica: 73b47622b2d240ce9200cb39d3ef5d0c, State: Running, Role: LEADER
I20260505 20:18:49.181344 31595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b47622b2d240ce9200cb39d3ef5d0c" member_type: VOTER }
I20260505 20:18:49.182121 31592 sys_catalog.cc:565] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:49.184588 31596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73b47622b2d240ce9200cb39d3ef5d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b47622b2d240ce9200cb39d3ef5d0c" member_type: VOTER } }
I20260505 20:18:49.184690 31597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73b47622b2d240ce9200cb39d3ef5d0c. Latest consensus state: current_term: 1 leader_uuid: "73b47622b2d240ce9200cb39d3ef5d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b47622b2d240ce9200cb39d3ef5d0c" member_type: VOTER } }
I20260505 20:18:49.185165 31596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:49.185240 31597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:49.190759 31600 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:49.199234 31600 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:49.201969 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:49.210526 31600 catalog_manager.cc:1357] Generated new cluster ID: 33b56566d1b0479998ef3b0245f3e071
I20260505 20:18:49.210852 31600 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:49.233619 31600 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:49.235030 31600 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:49.249336 31600 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c: Generated new TSK 0
I20260505 20:18:49.250061 31600 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:49.268962 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:49.276573 31613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:49.277114 31614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:49.280596 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:49.280864 31616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:49.281981 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:49.282227 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:49.282408 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012329282387 us; error 0 us; skew 500 ppm
I20260505 20:18:49.283097 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:49.286006 28669 webserver.cc:492] Webserver started at http://127.27.255.65:39085/ using document root <none> and password file <none>
I20260505 20:18:49.286512 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:49.286718 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:49.286999 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:49.288203 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "89da872a35134d29a0fd19524e5dde5b"
format_stamp: "Formatted at 2026-05-05 20:18:49 on dist-test-slave-dnnj"
I20260505 20:18:49.293254 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260505 20:18:49.296912 31621 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:49.297930 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260505 20:18:49.298269 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "89da872a35134d29a0fd19524e5dde5b"
format_stamp: "Formatted at 2026-05-05 20:18:49 on dist-test-slave-dnnj"
I20260505 20:18:49.298575 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:49.321724 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:49.322783 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:49.341331 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:49.341704 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260505 20:18:49.341996 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:49.342206 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:49.417604 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:34411
I20260505 20:18:49.417703 31691 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:34411 every 8 connection(s)
I20260505 20:18:49.422837 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260505 20:18:49.422947 31692 heartbeater.cc:344] Connected to a master server at 127.27.255.126:35053
I20260505 20:18:49.423417 31692 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:49.424335 31692 heartbeater.cc:507] Master 127.27.255.126:35053 requested a full tablet report, sending...
I20260505 20:18:49.426761 31557 ts_manager.cc:194] Registered new tserver with Master: 89da872a35134d29a0fd19524e5dde5b (127.27.255.65:34411)
W20260505 20:18:49.430020 31694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:49.430279 31557 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39128
W20260505 20:18:49.435628 31695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:49.440034 31697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:49.443794 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:49.445384 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:49.445664 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:49.445925 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012329445907 us; error 0 us; skew 500 ppm
I20260505 20:18:49.446453 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:49.449445 28669 webserver.cc:492] Webserver started at http://127.27.255.66:45511/ using document root <none> and password file <none>
I20260505 20:18:49.450085 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:49.450326 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:49.450721 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:49.452126 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-1-root/instance:
uuid: "52a45203d55b43ff945e484c7bd8d124"
format_stamp: "Formatted at 2026-05-05 20:18:49 on dist-test-slave-dnnj"
I20260505 20:18:49.457950 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260505 20:18:49.461961 31702 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:49.463045 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260505 20:18:49.463418 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-1-root
uuid: "52a45203d55b43ff945e484c7bd8d124"
format_stamp: "Formatted at 2026-05-05 20:18:49 on dist-test-slave-dnnj"
I20260505 20:18:49.463747 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:49.482954 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:49.484066 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:49.490079 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:49.490522 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260505 20:18:49.490947 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:49.491292 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:49.567828 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.66:33703
I20260505 20:18:49.567938 31771 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.66:33703 every 8 connection(s)
I20260505 20:18:49.573820 31772 heartbeater.cc:344] Connected to a master server at 127.27.255.126:35053
I20260505 20:18:49.574265 31772 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:49.575039 31772 heartbeater.cc:507] Master 127.27.255.126:35053 requested a full tablet report, sending...
I20260505 20:18:49.577360 31557 ts_manager.cc:194] Registered new tserver with Master: 52a45203d55b43ff945e484c7bd8d124 (127.27.255.66:33703)
I20260505 20:18:49.578307 28669 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.006906858s
I20260505 20:18:49.578848 31557 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39148
I20260505 20:18:49.619280 31657 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260505 20:18:49.620544 31657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:49.632789 31782 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: Bootstrap starting.
I20260505 20:18:49.639261 31782 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:49.644438 31782 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: No bootstrap required, opened a new log
I20260505 20:18:49.644940 31782 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260505 20:18:49.647774 31782 raft_consensus.cc:359] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } }
I20260505 20:18:49.648331 31782 raft_consensus.cc:385] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:49.648617 31782 raft_consensus.cc:740] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89da872a35134d29a0fd19524e5dde5b, State: Initialized, Role: FOLLOWER
I20260505 20:18:49.649269 31782 consensus_queue.cc:260] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } }
I20260505 20:18:49.649936 31782 raft_consensus.cc:399] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:49.650252 31782 raft_consensus.cc:493] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:49.650570 31782 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:49.657603 31782 raft_consensus.cc:515] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } }
I20260505 20:18:49.658375 31782 leader_election.cc:304] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89da872a35134d29a0fd19524e5dde5b; no voters:
I20260505 20:18:49.660037 31782 leader_election.cc:290] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:49.660447 31784 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:49.662281 31692 heartbeater.cc:499] Master 127.27.255.126:35053 was elected leader, sending a full tablet report...
I20260505 20:18:49.662950 31782 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: Time spent starting tablet: real 0.018s user 0.018s sys 0.001s
I20260505 20:18:49.663374 31784 raft_consensus.cc:697] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 1 LEADER]: Becoming Leader. State: Replica: 89da872a35134d29a0fd19524e5dde5b, State: Running, Role: LEADER
I20260505 20:18:49.664031 31784 consensus_queue.cc:237] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } }
I20260505 20:18:49.669118 31785 tablet_replica.cc:442] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89da872a35134d29a0fd19524e5dde5b" committed_config { opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } } }
I20260505 20:18:49.670086 31785 tablet_replica.cc:445] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: This TxnStatusTablet replica's current role is: LEADER
I20260505 20:18:49.671680 31786 tablet_replica.cc:442] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: TxnStatusTablet state changed. Reason: New leader 89da872a35134d29a0fd19524e5dde5b. Latest consensus state: current_term: 1 leader_uuid: "89da872a35134d29a0fd19524e5dde5b" committed_config { opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } } }
I20260505 20:18:49.672338 31786 tablet_replica.cc:445] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: This TxnStatusTablet replica's current role is: LEADER
W20260505 20:18:49.672991 31557 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:49.677520 31789 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260505 20:18:49.677958 31789 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260505 20:18:49.680318 31557 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:49.735210 31793 ts_tablet_manager.cc:933] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: Initiating tablet copy from peer 89da872a35134d29a0fd19524e5dde5b (127.27.255.65:34411)
I20260505 20:18:49.737048 31793 tablet_copy_client.cc:323] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: tablet copy: Beginning tablet copy session from remote peer at address 127.27.255.65:34411
I20260505 20:18:49.745863 31677 tablet_copy_service.cc:140] P 89da872a35134d29a0fd19524e5dde5b: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 52a45203d55b43ff945e484c7bd8d124 ({username='slave'} at 127.0.0.1:45922)
I20260505 20:18:49.746447 31677 tablet_copy_service.cc:161] P 89da872a35134d29a0fd19524e5dde5b: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 52a45203d55b43ff945e484c7bd8d124 at {username='slave'} at 127.0.0.1:45922: session id = 52a45203d55b43ff945e484c7bd8d124-11111111111111111111111111111111
I20260505 20:18:49.751792 31677 tablet_copy_source_session.cc:215] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: Tablet Copy: opened 0 blocks and 1 log segments
I20260505 20:18:49.754865 31793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:49.766076 31793 tablet_copy_client.cc:806] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: tablet copy: Starting download of 0 data blocks...
I20260505 20:18:49.766728 31793 tablet_copy_client.cc:670] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: tablet copy: Starting download of 1 WAL segments...
I20260505 20:18:49.770807 31793 tablet_copy_client.cc:538] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260505 20:18:49.777036 31793 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: Bootstrap starting.
I20260505 20:18:49.802448 31793 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 20:18:49.803231 31793 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: Bootstrap complete.
I20260505 20:18:49.803799 31793 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.009s
I20260505 20:18:49.805898 31793 raft_consensus.cc:359] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } }
I20260505 20:18:49.806329 31793 raft_consensus.cc:740] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 52a45203d55b43ff945e484c7bd8d124, State: Initialized, Role: NON_PARTICIPANT
I20260505 20:18:49.806859 31793 consensus_queue.cc:260] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } }
I20260505 20:18:49.808646 31796 tablet_replica.cc:442] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } } }
I20260505 20:18:49.808828 31772 heartbeater.cc:499] Master 127.27.255.126:35053 was elected leader, sending a full tablet report...
I20260505 20:18:49.809136 31793 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260505 20:18:49.809199 31796 tablet_replica.cc:445] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20260505 20:18:49.811275 31677 tablet_copy_service.cc:342] P 89da872a35134d29a0fd19524e5dde5b: Request end of tablet copy session 52a45203d55b43ff945e484c7bd8d124-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:45922
I20260505 20:18:49.811774 31677 tablet_copy_service.cc:434] P 89da872a35134d29a0fd19524e5dde5b: ending tablet copy session 52a45203d55b43ff945e484c7bd8d124-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 52a45203d55b43ff945e484c7bd8d124
W20260505 20:18:49.811870 31557 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260505 20:18:49.816236 31557 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:49.831323 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:49.862231 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:49.863358 28669 tablet_replica.cc:333] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: stopping tablet replica
I20260505 20:18:49.863853 28669 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:49.864326 28669 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:49.888980 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:49.901594 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:49.907859 31797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:49.909538 31798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:49.912078 31800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:49.918092 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:49.919340 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:49.919879 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:49.920099 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012329920080 us; error 0 us; skew 500 ppm
I20260505 20:18:49.921025 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:49.924336 28669 webserver.cc:492] Webserver started at http://127.27.255.65:39085/ using document root <none> and password file <none>
I20260505 20:18:49.925048 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:49.925287 28669 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260505 20:18:49.930231 28669 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260505 20:18:49.934648 31805 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:49.935688 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260505 20:18:49.936028 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "89da872a35134d29a0fd19524e5dde5b"
format_stamp: "Formatted at 2026-05-05 20:18:49 on dist-test-slave-dnnj"
I20260505 20:18:49.936374 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:49.958294 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:49.959551 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:49.967768 31814 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20260505 20:18:49.976310 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260505 20:18:49.976761 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.001s
I20260505 20:18:49.977161 28669 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260505 20:18:49.985024 28669 ts_tablet_manager.cc:616] Registered 1 tablets
I20260505 20:18:49.985337 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.008s user 0.004s sys 0.004s
I20260505 20:18:49.985368 31814 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: Bootstrap starting.
I20260505 20:18:50.019044 31814 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 20:18:50.020344 31814 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: Bootstrap complete.
I20260505 20:18:50.021186 31814 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: Time spent bootstrapping tablet: real 0.036s user 0.021s sys 0.010s
I20260505 20:18:50.024084 31814 raft_consensus.cc:359] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } }
I20260505 20:18:50.024665 31814 raft_consensus.cc:740] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89da872a35134d29a0fd19524e5dde5b, State: Initialized, Role: FOLLOWER
I20260505 20:18:50.025322 31814 consensus_queue.cc:260] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } }
I20260505 20:18:50.025971 31814 raft_consensus.cc:399] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:50.026252 31814 raft_consensus.cc:493] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:50.026589 31814 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:50.033718 31814 raft_consensus.cc:515] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } }
I20260505 20:18:50.034483 31814 leader_election.cc:304] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89da872a35134d29a0fd19524e5dde5b; no voters:
I20260505 20:18:50.035996 31814 leader_election.cc:290] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260505 20:18:50.036517 31837 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 2 FOLLOWER]: Leader election won for term 2
I20260505 20:18:50.045300 31837 raft_consensus.cc:697] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 2 LEADER]: Becoming Leader. State: Replica: 89da872a35134d29a0fd19524e5dde5b, State: Running, Role: LEADER
I20260505 20:18:50.045948 31814 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b: Time spent starting tablet: real 0.024s user 0.018s sys 0.005s
I20260505 20:18:50.047098 31837 consensus_queue.cc:237] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } }
I20260505 20:18:50.050632 31838 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "89da872a35134d29a0fd19524e5dde5b" committed_config { opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } } }
I20260505 20:18:50.051339 31838 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260505 20:18:50.056284 31839 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 89da872a35134d29a0fd19524e5dde5b. Latest consensus state: current_term: 2 leader_uuid: "89da872a35134d29a0fd19524e5dde5b" committed_config { opid_index: -1 peers { permanent_uuid: "89da872a35134d29a0fd19524e5dde5b" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 34411 } } }
I20260505 20:18:50.057071 31839 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260505 20:18:50.063568 31846 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260505 20:18:50.064069 31846 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260505 20:18:50.102806 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:34411
I20260505 20:18:50.102893 31883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:34411 every 8 connection(s)
I20260505 20:18:50.108884 31884 heartbeater.cc:344] Connected to a master server at 127.27.255.126:35053
I20260505 20:18:50.109468 31884 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:50.110528 31884 heartbeater.cc:507] Master 127.27.255.126:35053 requested a full tablet report, sending...
I20260505 20:18:50.111086 28669 tablet_server.cc:179] TabletServer@127.27.255.65:34411 shutting down...
I20260505 20:18:50.115331 31557 ts_manager.cc:194] Re-registered known tserver with Master: 89da872a35134d29a0fd19524e5dde5b (127.27.255.65:34411)
W20260505 20:18:50.116255 31557 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:50.118608 31557 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39178
I20260505 20:18:50.122750 31884 heartbeater.cc:499] Master 127.27.255.126:35053 was elected leader, sending a full tablet report...
W20260505 20:18:50.126663 31557 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:50.146771 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:50.147879 28669 tablet_replica.cc:333] stopping tablet replica
I20260505 20:18:50.148481 28669 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 2 LEADER]: Raft consensus shutting down.
I20260505 20:18:50.149013 28669 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 89da872a35134d29a0fd19524e5dde5b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:50.162024 28669 tablet_server.cc:196] TabletServer@127.27.255.65:34411 shutdown complete.
I20260505 20:18:50.175899 28669 tablet_server.cc:179] TabletServer@127.27.255.66:0 shutting down...
I20260505 20:18:50.202565 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:50.203769 28669 tablet_replica.cc:333] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124: stopping tablet replica
I20260505 20:18:50.204330 28669 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260505 20:18:50.204731 28669 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 52a45203d55b43ff945e484c7bd8d124 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260505 20:18:50.224462 28669 tablet_server.cc:196] TabletServer@127.27.255.66:0 shutdown complete.
I20260505 20:18:50.236760 28669 master.cc:562] Master@127.27.255.126:35053 shutting down...
I20260505 20:18:50.258477 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:50.259085 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:50.259425 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P 73b47622b2d240ce9200cb39d3ef5d0c: stopping tablet replica
I20260505 20:18:50.278529 28669 master.cc:584] Master@127.27.255.126:35053 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1256 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20260505 20:18:50.305215 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:32941
I20260505 20:18:50.306326 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:50.312386 31891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:50.312358 31890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:50.315702 31893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:50.316171 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:50.317086 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:50.317256 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:50.317441 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012330317425 us; error 0 us; skew 500 ppm
I20260505 20:18:50.318039 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:50.320897 28669 webserver.cc:492] Webserver started at http://127.27.255.126:38283/ using document root <none> and password file <none>
I20260505 20:18:50.321405 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:50.321750 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:50.322024 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:50.323470 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "686eedd334624b45af92214fe9ab254a"
format_stamp: "Formatted at 2026-05-05 20:18:50 on dist-test-slave-dnnj"
I20260505 20:18:50.328195 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260505 20:18:50.332970 31898 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:50.333887 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260505 20:18:50.334178 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "686eedd334624b45af92214fe9ab254a"
format_stamp: "Formatted at 2026-05-05 20:18:50 on dist-test-slave-dnnj"
I20260505 20:18:50.334446 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:50.366401 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:50.367569 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:50.421561 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:32941
I20260505 20:18:50.421797 31949 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:32941 every 8 connection(s)
I20260505 20:18:50.426128 31950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:50.443060 31950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a: Bootstrap starting.
I20260505 20:18:50.447808 31950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:50.452243 31950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a: No bootstrap required, opened a new log
I20260505 20:18:50.455227 31950 raft_consensus.cc:359] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686eedd334624b45af92214fe9ab254a" member_type: VOTER }
I20260505 20:18:50.455714 31950 raft_consensus.cc:385] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:50.455987 31950 raft_consensus.cc:740] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686eedd334624b45af92214fe9ab254a, State: Initialized, Role: FOLLOWER
I20260505 20:18:50.456645 31950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686eedd334624b45af92214fe9ab254a" member_type: VOTER }
I20260505 20:18:50.457130 31950 raft_consensus.cc:399] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:50.457383 31950 raft_consensus.cc:493] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:50.457774 31950 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:50.462644 31950 raft_consensus.cc:515] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686eedd334624b45af92214fe9ab254a" member_type: VOTER }
I20260505 20:18:50.463212 31950 leader_election.cc:304] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 686eedd334624b45af92214fe9ab254a; no voters:
I20260505 20:18:50.464579 31950 leader_election.cc:290] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:50.464915 31953 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:50.466651 31953 raft_consensus.cc:697] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 1 LEADER]: Becoming Leader. State: Replica: 686eedd334624b45af92214fe9ab254a, State: Running, Role: LEADER
I20260505 20:18:50.467432 31953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686eedd334624b45af92214fe9ab254a" member_type: VOTER }
I20260505 20:18:50.468075 31950 sys_catalog.cc:565] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:50.470562 31954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "686eedd334624b45af92214fe9ab254a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686eedd334624b45af92214fe9ab254a" member_type: VOTER } }
I20260505 20:18:50.470659 31955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 686eedd334624b45af92214fe9ab254a. Latest consensus state: current_term: 1 leader_uuid: "686eedd334624b45af92214fe9ab254a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "686eedd334624b45af92214fe9ab254a" member_type: VOTER } }
I20260505 20:18:50.471144 31954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:50.471306 31955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:50.485327 31958 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:50.493399 31958 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:50.495059 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:50.504896 31958 catalog_manager.cc:1357] Generated new cluster ID: bf28be2962784805b72ade22268697f9
I20260505 20:18:50.505185 31958 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:50.530809 31958 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:50.532188 31958 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:50.547224 31958 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a: Generated new TSK 0
I20260505 20:18:50.547857 31958 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:50.562376 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:50.569881 31971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:50.573192 31972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:50.574615 31974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:50.576076 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:50.577106 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:50.577368 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:50.577605 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012330577577 us; error 0 us; skew 500 ppm
I20260505 20:18:50.578318 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:50.581776 28669 webserver.cc:492] Webserver started at http://127.27.255.65:44605/ using document root <none> and password file <none>
I20260505 20:18:50.582415 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:50.582659 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:50.583012 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:50.584465 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "47e396b3420b485d863cd2b0f6431e74"
format_stamp: "Formatted at 2026-05-05 20:18:50 on dist-test-slave-dnnj"
I20260505 20:18:50.590797 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260505 20:18:50.595672 31979 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:50.596676 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260505 20:18:50.597030 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "47e396b3420b485d863cd2b0f6431e74"
format_stamp: "Formatted at 2026-05-05 20:18:50 on dist-test-slave-dnnj"
I20260505 20:18:50.597398 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:50.610597 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:50.611986 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:50.638129 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:50.638463 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:50.638787 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:50.638988 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:50.715518 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:36389
I20260505 20:18:50.715696 32049 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:36389 every 8 connection(s)
I20260505 20:18:50.720564 32050 heartbeater.cc:344] Connected to a master server at 127.27.255.126:32941
I20260505 20:18:50.721031 32050 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:50.722028 32050 heartbeater.cc:507] Master 127.27.255.126:32941 requested a full tablet report, sending...
I20260505 20:18:50.724076 31915 ts_manager.cc:194] Registered new tserver with Master: 47e396b3420b485d863cd2b0f6431e74 (127.27.255.65:36389)
I20260505 20:18:50.724972 28669 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.006058516s
I20260505 20:18:50.726027 31915 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40640
I20260505 20:18:50.761449 32013 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260505 20:18:50.762717 32013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:50.774801 32060 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74: Bootstrap starting.
I20260505 20:18:50.780097 32060 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:50.785192 32060 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74: No bootstrap required, opened a new log
I20260505 20:18:50.785676 32060 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260505 20:18:50.787649 32060 raft_consensus.cc:359] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "47e396b3420b485d863cd2b0f6431e74" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36389 } }
I20260505 20:18:50.788086 32060 raft_consensus.cc:385] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:50.788321 32060 raft_consensus.cc:740] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e396b3420b485d863cd2b0f6431e74, State: Initialized, Role: FOLLOWER
I20260505 20:18:50.789222 32060 consensus_queue.cc:260] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "47e396b3420b485d863cd2b0f6431e74" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36389 } }
I20260505 20:18:50.789800 32060 raft_consensus.cc:399] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:50.790434 32060 raft_consensus.cc:493] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:50.790752 32060 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:50.795574 32060 raft_consensus.cc:515] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "47e396b3420b485d863cd2b0f6431e74" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36389 } }
I20260505 20:18:50.796169 32060 leader_election.cc:304] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47e396b3420b485d863cd2b0f6431e74; no voters:
I20260505 20:18:50.798008 32060 leader_election.cc:290] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:50.798571 32062 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:50.800042 32050 heartbeater.cc:499] Master 127.27.255.126:32941 was elected leader, sending a full tablet report...
I20260505 20:18:50.801970 32060 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260505 20:18:50.801795 32062 raft_consensus.cc:697] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 1 LEADER]: Becoming Leader. State: Replica: 47e396b3420b485d863cd2b0f6431e74, State: Running, Role: LEADER
I20260505 20:18:50.802718 32062 consensus_queue.cc:237] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "47e396b3420b485d863cd2b0f6431e74" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36389 } }
I20260505 20:18:50.805670 32063 tablet_replica.cc:442] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47e396b3420b485d863cd2b0f6431e74" committed_config { opid_index: -1 peers { permanent_uuid: "47e396b3420b485d863cd2b0f6431e74" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36389 } } }
I20260505 20:18:50.806229 32063 tablet_replica.cc:445] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74: This TxnStatusTablet replica's current role is: LEADER
W20260505 20:18:50.809588 31915 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:50.814879 32066 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260505 20:18:50.815402 32066 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260505 20:18:50.816073 32064 tablet_replica.cc:442] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74: TxnStatusTablet state changed. Reason: New leader 47e396b3420b485d863cd2b0f6431e74. Latest consensus state: current_term: 1 leader_uuid: "47e396b3420b485d863cd2b0f6431e74" committed_config { opid_index: -1 peers { permanent_uuid: "47e396b3420b485d863cd2b0f6431e74" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 36389 } } }
I20260505 20:18:50.816666 32064 tablet_replica.cc:445] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74: This TxnStatusTablet replica's current role is: LEADER
W20260505 20:18:50.818153 31915 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260505 20:18:50.871886 32009 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260505 20:18:50.872242 32009 txn_status_manager.cc:244] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260505 20:18:50.872680 32009 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20260505 20:18:50.881994 32009 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260505 20:18:50.924752 32009 txn_status_manager.cc:244] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260505 20:18:50.925083 32009 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20260505 20:18:50.935442 32009 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20260505 20:18:50.947530 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:35282
W20260505 20:18:50.949911 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:35282
W20260505 20:18:50.952107 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:35282
W20260505 20:18:50.954332 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:35282
W20260505 20:18:50.956763 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:35282
W20260505 20:18:50.960431 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:35282
W20260505 20:18:50.973826 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:35298
W20260505 20:18:50.976080 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:35298
W20260505 20:18:50.978361 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:35298
W20260505 20:18:50.980664 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:35298
W20260505 20:18:50.982826 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:35298
W20260505 20:18:50.985172 32013 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:35298
I20260505 20:18:50.988387 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:51.020350 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:51.021319 28669 tablet_replica.cc:333] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74: stopping tablet replica
I20260505 20:18:51.022006 28669 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:51.022622 28669 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 47e396b3420b485d863cd2b0f6431e74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:51.037396 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:51.050397 28669 master.cc:562] Master@127.27.255.126:32941 shutting down...
I20260505 20:18:51.073654 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:51.074304 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:51.074813 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P 686eedd334624b45af92214fe9ab254a: stopping tablet replica
I20260505 20:18:51.096395 28669 master.cc:584] Master@127.27.255.126:32941 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (821 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20260505 20:18:51.127385 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:38145
I20260505 20:18:51.128482 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:51.134202 32075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:51.134275 32076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:51.138355 32078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:51.138939 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:51.139704 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:51.139871 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:51.140057 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012331140024 us; error 0 us; skew 500 ppm
I20260505 20:18:51.140753 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:51.143882 28669 webserver.cc:492] Webserver started at http://127.27.255.126:41845/ using document root <none> and password file <none>
I20260505 20:18:51.144369 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:51.144560 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:51.144840 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:51.146397 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "6d1ae789b6984aba96febb91753613fb"
format_stamp: "Formatted at 2026-05-05 20:18:51 on dist-test-slave-dnnj"
I20260505 20:18:51.152544 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260505 20:18:51.156490 32083 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:51.157620 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260505 20:18:51.158143 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "6d1ae789b6984aba96febb91753613fb"
format_stamp: "Formatted at 2026-05-05 20:18:51 on dist-test-slave-dnnj"
I20260505 20:18:51.158679 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:51.184442 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:51.185417 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:51.222005 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:38145
I20260505 20:18:51.222113 32134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:38145 every 8 connection(s)
I20260505 20:18:51.226078 32135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:51.236512 32135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb: Bootstrap starting.
I20260505 20:18:51.240839 32135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:51.245054 32135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb: No bootstrap required, opened a new log
I20260505 20:18:51.247195 32135 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1ae789b6984aba96febb91753613fb" member_type: VOTER }
I20260505 20:18:51.247581 32135 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:51.247855 32135 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1ae789b6984aba96febb91753613fb, State: Initialized, Role: FOLLOWER
I20260505 20:18:51.248597 32135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1ae789b6984aba96febb91753613fb" member_type: VOTER }
I20260505 20:18:51.249046 32135 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:51.249259 32135 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:51.249605 32135 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:51.254139 32135 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1ae789b6984aba96febb91753613fb" member_type: VOTER }
I20260505 20:18:51.254626 32135 leader_election.cc:304] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d1ae789b6984aba96febb91753613fb; no voters:
I20260505 20:18:51.255779 32135 leader_election.cc:290] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:51.256125 32138 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:51.257670 32138 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 1 LEADER]: Becoming Leader. State: Replica: 6d1ae789b6984aba96febb91753613fb, State: Running, Role: LEADER
I20260505 20:18:51.258337 32138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1ae789b6984aba96febb91753613fb" member_type: VOTER }
I20260505 20:18:51.258910 32135 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:51.261566 32139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d1ae789b6984aba96febb91753613fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1ae789b6984aba96febb91753613fb" member_type: VOTER } }
I20260505 20:18:51.261706 32140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1ae789b6984aba96febb91753613fb. Latest consensus state: current_term: 1 leader_uuid: "6d1ae789b6984aba96febb91753613fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1ae789b6984aba96febb91753613fb" member_type: VOTER } }
I20260505 20:18:51.262516 32140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:51.262486 32139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:51.278981 32143 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:51.284842 32143 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:51.288793 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:51.294425 32143 catalog_manager.cc:1357] Generated new cluster ID: a94919737678489cad577b190539623d
I20260505 20:18:51.294771 32143 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:51.331722 32143 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:51.333005 32143 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:51.361739 32143 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb: Generated new TSK 0
I20260505 20:18:51.362337 32143 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:51.420383 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:51.427762 32156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:51.428244 32157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:51.430347 32159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:51.431991 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:51.432931 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:51.433123 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:51.433264 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012331433251 us; error 0 us; skew 500 ppm
I20260505 20:18:51.433853 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:51.436365 28669 webserver.cc:492] Webserver started at http://127.27.255.65:40879/ using document root <none> and password file <none>
I20260505 20:18:51.436841 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:51.436997 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:51.437212 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:51.438401 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "da622beabf6b4cd3b7891df7b99a463a"
format_stamp: "Formatted at 2026-05-05 20:18:51 on dist-test-slave-dnnj"
I20260505 20:18:51.443739 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260505 20:18:51.447494 32164 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:51.448741 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:51.449155 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "da622beabf6b4cd3b7891df7b99a463a"
format_stamp: "Formatted at 2026-05-05 20:18:51 on dist-test-slave-dnnj"
I20260505 20:18:51.449769 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:51.461118 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:51.462235 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:51.467499 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:51.467873 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260505 20:18:51.468356 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:51.468611 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:51.527486 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:39673
I20260505 20:18:51.527593 32234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:39673 every 8 connection(s)
I20260505 20:18:51.532325 32235 heartbeater.cc:344] Connected to a master server at 127.27.255.126:38145
I20260505 20:18:51.532827 32235 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:51.533844 32235 heartbeater.cc:507] Master 127.27.255.126:38145 requested a full tablet report, sending...
I20260505 20:18:51.535924 32100 ts_manager.cc:194] Registered new tserver with Master: da622beabf6b4cd3b7891df7b99a463a (127.27.255.65:39673)
I20260505 20:18:51.536041 28669 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.005406739s
I20260505 20:18:51.537791 32100 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45558
I20260505 20:18:51.574684 32200 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260505 20:18:51.575968 32200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:51.589259 32245 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a: Bootstrap starting.
I20260505 20:18:51.594084 32245 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:51.598420 32245 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a: No bootstrap required, opened a new log
I20260505 20:18:51.598835 32245 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260505 20:18:51.600726 32245 raft_consensus.cc:359] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "da622beabf6b4cd3b7891df7b99a463a" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 39673 } }
I20260505 20:18:51.601109 32245 raft_consensus.cc:385] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:51.601358 32245 raft_consensus.cc:740] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da622beabf6b4cd3b7891df7b99a463a, State: Initialized, Role: FOLLOWER
I20260505 20:18:51.601985 32245 consensus_queue.cc:260] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "da622beabf6b4cd3b7891df7b99a463a" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 39673 } }
I20260505 20:18:51.602438 32245 raft_consensus.cc:399] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:51.602660 32245 raft_consensus.cc:493] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:51.602958 32245 raft_consensus.cc:3060] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:51.607487 32245 raft_consensus.cc:515] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "da622beabf6b4cd3b7891df7b99a463a" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 39673 } }
I20260505 20:18:51.608021 32245 leader_election.cc:304] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da622beabf6b4cd3b7891df7b99a463a; no voters:
I20260505 20:18:51.609186 32245 leader_election.cc:290] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:51.609664 32247 raft_consensus.cc:2804] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:51.611295 32247 raft_consensus.cc:697] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 1 LEADER]: Becoming Leader. State: Replica: da622beabf6b4cd3b7891df7b99a463a, State: Running, Role: LEADER
I20260505 20:18:51.611866 32235 heartbeater.cc:499] Master 127.27.255.126:38145 was elected leader, sending a full tablet report...
I20260505 20:18:51.612078 32247 consensus_queue.cc:237] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "da622beabf6b4cd3b7891df7b99a463a" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 39673 } }
I20260505 20:18:51.612310 32245 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260505 20:18:51.616163 32249 tablet_replica.cc:442] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a: TxnStatusTablet state changed. Reason: New leader da622beabf6b4cd3b7891df7b99a463a. Latest consensus state: current_term: 1 leader_uuid: "da622beabf6b4cd3b7891df7b99a463a" committed_config { opid_index: -1 peers { permanent_uuid: "da622beabf6b4cd3b7891df7b99a463a" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 39673 } } }
I20260505 20:18:51.616530 32248 tablet_replica.cc:442] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da622beabf6b4cd3b7891df7b99a463a" committed_config { opid_index: -1 peers { permanent_uuid: "da622beabf6b4cd3b7891df7b99a463a" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 39673 } } }
I20260505 20:18:51.616781 32249 tablet_replica.cc:445] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a: This TxnStatusTablet replica's current role is: LEADER
I20260505 20:18:51.617148 32248 tablet_replica.cc:445] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a: This TxnStatusTablet replica's current role is: LEADER
W20260505 20:18:51.619201 32100 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:51.619297 32252 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260505 20:18:51.619783 32252 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260505 20:18:51.623783 32100 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260505 20:18:51.648449 32200 tablet_service.cc:1511] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260505 20:18:51.649730 32200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:51.661018 32245 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a: Bootstrap starting.
I20260505 20:18:51.665304 32245 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:51.669098 32245 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a: No bootstrap required, opened a new log
I20260505 20:18:51.669692 32245 ts_tablet_manager.cc:1403] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260505 20:18:51.671624 32245 raft_consensus.cc:359] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "da622beabf6b4cd3b7891df7b99a463a" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 39673 } }
I20260505 20:18:51.672091 32245 raft_consensus.cc:385] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:51.672327 32245 raft_consensus.cc:740] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da622beabf6b4cd3b7891df7b99a463a, State: Initialized, Role: FOLLOWER
I20260505 20:18:51.672825 32245 consensus_queue.cc:260] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "da622beabf6b4cd3b7891df7b99a463a" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 39673 } }
I20260505 20:18:51.673276 32245 raft_consensus.cc:399] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:51.673550 32245 raft_consensus.cc:493] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:51.673827 32245 raft_consensus.cc:3060] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:51.678598 32245 raft_consensus.cc:515] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "da622beabf6b4cd3b7891df7b99a463a" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 39673 } }
I20260505 20:18:51.679230 32245 leader_election.cc:304] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da622beabf6b4cd3b7891df7b99a463a; no voters:
I20260505 20:18:51.679821 32245 leader_election.cc:290] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:51.680006 32247 raft_consensus.cc:2804] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:51.680536 32247 raft_consensus.cc:697] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 1 LEADER]: Becoming Leader. State: Replica: da622beabf6b4cd3b7891df7b99a463a, State: Running, Role: LEADER
I20260505 20:18:51.681679 32245 ts_tablet_manager.cc:1434] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260505 20:18:51.681272 32247 consensus_queue.cc:237] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "da622beabf6b4cd3b7891df7b99a463a" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 39673 } }
W20260505 20:18:51.687476 32100 catalog_manager.cc:5504] Ignoring report from unknown tablet 22222222222222222222222222222222
W20260505 20:18:51.691672 32100 catalog_manager.cc:5504] Ignoring report from unknown tablet 22222222222222222222222222222222
I20260505 20:18:51.714218 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:51.741309 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:51.742168 28669 tablet_replica.cc:333] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a: stopping tablet replica
I20260505 20:18:51.742728 28669 raft_consensus.cc:2243] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:51.743167 28669 raft_consensus.cc:2272] T 22222222222222222222222222222222 P da622beabf6b4cd3b7891df7b99a463a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:51.746062 28669 tablet_replica.cc:333] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a: stopping tablet replica
I20260505 20:18:51.746575 28669 raft_consensus.cc:2243] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:51.747048 28669 raft_consensus.cc:2272] T 11111111111111111111111111111111 P da622beabf6b4cd3b7891df7b99a463a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:51.772305 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:51.784513 28669 master.cc:562] Master@127.27.255.126:38145 shutting down...
I20260505 20:18:51.803891 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:51.804548 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:51.805089 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P 6d1ae789b6984aba96febb91753613fb: stopping tablet replica
I20260505 20:18:51.824230 28669 master.cc:584] Master@127.27.255.126:38145 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (742 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3774 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20260505 20:18:51.868883 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:39763
I20260505 20:18:51.870061 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:51.875433 32261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:51.876144 32262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:51.879175 32264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:51.879189 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:51.880275 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:51.880488 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:51.880650 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012331880632 us; error 0 us; skew 500 ppm
I20260505 20:18:51.881201 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:51.883760 28669 webserver.cc:492] Webserver started at http://127.27.255.126:35319/ using document root <none> and password file <none>
I20260505 20:18:51.884254 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:51.884464 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:51.884747 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:51.885987 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "d3d2291c044040f4835225cf1659a9a9"
format_stamp: "Formatted at 2026-05-05 20:18:51 on dist-test-slave-dnnj"
I20260505 20:18:51.891080 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260505 20:18:51.895486 32269 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:51.896332 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260505 20:18:51.896736 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "d3d2291c044040f4835225cf1659a9a9"
format_stamp: "Formatted at 2026-05-05 20:18:51 on dist-test-slave-dnnj"
I20260505 20:18:51.897054 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:51.912541 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:51.913679 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:51.954798 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:39763
I20260505 20:18:51.954895 32320 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:39763 every 8 connection(s)
I20260505 20:18:51.959245 32321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:51.971692 32321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9: Bootstrap starting.
I20260505 20:18:51.976083 32321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:51.980139 32321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9: No bootstrap required, opened a new log
I20260505 20:18:51.982590 32321 raft_consensus.cc:359] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2291c044040f4835225cf1659a9a9" member_type: VOTER }
I20260505 20:18:51.983125 32321 raft_consensus.cc:385] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:51.983335 32321 raft_consensus.cc:740] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d2291c044040f4835225cf1659a9a9, State: Initialized, Role: FOLLOWER
I20260505 20:18:51.984488 32321 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2291c044040f4835225cf1659a9a9" member_type: VOTER }
I20260505 20:18:51.985020 32324 raft_consensus.cc:493] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:51.985569 32324 raft_consensus.cc:515] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2291c044040f4835225cf1659a9a9" member_type: VOTER }
I20260505 20:18:51.986225 32324 leader_election.cc:304] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d2291c044040f4835225cf1659a9a9; no voters:
I20260505 20:18:51.987927 32324 leader_election.cc:290] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260505 20:18:51.988197 32325 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260505 20:18:51.988636 32325 raft_consensus.cc:493] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:51.984995 32321 raft_consensus.cc:399] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:51.988943 32325 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:51.994361 32325 raft_consensus.cc:515] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2291c044040f4835225cf1659a9a9" member_type: VOTER }
I20260505 20:18:51.994985 32325 leader_election.cc:304] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d2291c044040f4835225cf1659a9a9; no voters:
I20260505 20:18:51.995092 32321 raft_consensus.cc:493] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:51.995564 32325 leader_election.cc:290] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:51.995884 32321 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:18:52.001137 32321 raft_consensus.cc:515] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2291c044040f4835225cf1659a9a9" member_type: VOTER }
I20260505 20:18:52.001819 32324 raft_consensus.cc:2764] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 2 FOLLOWER]: Leader election decision vote started in defunct term 1: won
I20260505 20:18:52.002895 32321 leader_election.cc:304] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d2291c044040f4835225cf1659a9a9; no voters:
I20260505 20:18:52.003579 32321 leader_election.cc:290] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260505 20:18:52.003718 32325 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 2 FOLLOWER]: Leader election won for term 2
I20260505 20:18:52.005056 32325 raft_consensus.cc:697] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 2 LEADER]: Becoming Leader. State: Replica: d3d2291c044040f4835225cf1659a9a9, State: Running, Role: LEADER
I20260505 20:18:52.005721 32325 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2291c044040f4835225cf1659a9a9" member_type: VOTER }
I20260505 20:18:52.005900 32321 sys_catalog.cc:565] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:52.009646 32324 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d3d2291c044040f4835225cf1659a9a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2291c044040f4835225cf1659a9a9" member_type: VOTER } }
I20260505 20:18:52.010190 32324 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:52.010308 32326 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d2291c044040f4835225cf1659a9a9. Latest consensus state: current_term: 2 leader_uuid: "d3d2291c044040f4835225cf1659a9a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d2291c044040f4835225cf1659a9a9" member_type: VOTER } }
I20260505 20:18:52.011013 32326 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:52.014031 32330 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:52.021932 32330 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:52.023051 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:52.031527 32330 catalog_manager.cc:1357] Generated new cluster ID: c7941ad9cbc34d82b0e9dd9dfa449d81
I20260505 20:18:52.031872 32330 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:52.048013 32330 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:52.049331 32330 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:52.062402 32330 catalog_manager.cc:6044] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9: Generated new TSK 0
I20260505 20:18:52.063064 32330 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:52.090528 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:52.097029 32342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:52.098354 32343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:52.100147 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:52.100028 32345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:52.101845 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:52.102074 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:52.102243 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012332102220 us; error 0 us; skew 500 ppm
I20260505 20:18:52.102774 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:52.105085 28669 webserver.cc:492] Webserver started at http://127.27.255.65:41547/ using document root <none> and password file <none>
I20260505 20:18:52.105651 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:52.105863 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:52.106132 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:52.107299 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "cc702e3c0f2e456a96beeafb1754bb58"
format_stamp: "Formatted at 2026-05-05 20:18:52 on dist-test-slave-dnnj"
I20260505 20:18:52.112497 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260505 20:18:52.116253 32350 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:52.117152 28669 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260505 20:18:52.117405 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "cc702e3c0f2e456a96beeafb1754bb58"
format_stamp: "Formatted at 2026-05-05 20:18:52 on dist-test-slave-dnnj"
I20260505 20:18:52.117897 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:52.141031 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:52.142125 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:52.143669 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:52.146041 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:52.146216 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:52.146524 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:52.146688 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:52.193861 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:40229
I20260505 20:18:52.193926 32412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:40229 every 8 connection(s)
I20260505 20:18:52.199095 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:52.206396 32417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:52.208045 32418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:52.210129 32420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:52.211766 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:52.212940 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:52.213193 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:52.213398 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012332213378 us; error 0 us; skew 500 ppm
I20260505 20:18:52.213966 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:52.214493 32413 heartbeater.cc:344] Connected to a master server at 127.27.255.126:39763
I20260505 20:18:52.214994 32413 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:52.215739 32413 heartbeater.cc:507] Master 127.27.255.126:39763 requested a full tablet report, sending...
I20260505 20:18:52.216802 28669 webserver.cc:492] Webserver started at http://127.27.255.66:43853/ using document root <none> and password file <none>
I20260505 20:18:52.217403 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:52.217675 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:52.217945 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:52.218146 32286 ts_manager.cc:194] Registered new tserver with Master: cc702e3c0f2e456a96beeafb1754bb58 (127.27.255.65:40229)
I20260505 20:18:52.219079 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-1-root/instance:
uuid: "25a9103fcca94cadb1b50424e37aca85"
format_stamp: "Formatted at 2026-05-05 20:18:52 on dist-test-slave-dnnj"
I20260505 20:18:52.220685 32286 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60800
I20260505 20:18:52.224028 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260505 20:18:52.227686 32425 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:52.228645 28669 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260505 20:18:52.228907 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-1-root
uuid: "25a9103fcca94cadb1b50424e37aca85"
format_stamp: "Formatted at 2026-05-05 20:18:52 on dist-test-slave-dnnj"
I20260505 20:18:52.229151 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:52.244972 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:52.246134 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:52.248044 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:52.250458 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:52.250658 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:52.250839 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:52.250962 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:52.291147 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.66:40909
I20260505 20:18:52.291240 32487 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.66:40909 every 8 connection(s)
I20260505 20:18:52.295950 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:52.303263 32491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:52.303498 32492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:52.306852 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:52.307317 32495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:52.308133 32488 heartbeater.cc:344] Connected to a master server at 127.27.255.126:39763
I20260505 20:18:52.308504 32488 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:52.308589 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:52.308887 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:52.309052 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012332309020 us; error 0 us; skew 500 ppm
I20260505 20:18:52.309334 32488 heartbeater.cc:507] Master 127.27.255.126:39763 requested a full tablet report, sending...
I20260505 20:18:52.309726 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:52.311400 32286 ts_manager.cc:194] Registered new tserver with Master: 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66:40909)
I20260505 20:18:52.312444 28669 webserver.cc:492] Webserver started at http://127.27.255.67:34667/ using document root <none> and password file <none>
I20260505 20:18:52.313079 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:52.313247 32286 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60814
I20260505 20:18:52.313338 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:52.313892 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:52.315094 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-2-root/instance:
uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d"
format_stamp: "Formatted at 2026-05-05 20:18:52 on dist-test-slave-dnnj"
I20260505 20:18:52.319630 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260505 20:18:52.324313 32499 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:52.325480 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260505 20:18:52.325927 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-2-root
uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d"
format_stamp: "Formatted at 2026-05-05 20:18:52 on dist-test-slave-dnnj"
I20260505 20:18:52.326256 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:52.379850 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:52.380985 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:52.382582 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:52.384812 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:52.385057 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:52.385303 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:52.385478 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:52.432593 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.67:37321
I20260505 20:18:52.432690 32561 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.67:37321 every 8 connection(s)
I20260505 20:18:52.448846 32562 heartbeater.cc:344] Connected to a master server at 127.27.255.126:39763
I20260505 20:18:52.449215 32562 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:52.450580 32562 heartbeater.cc:507] Master 127.27.255.126:39763 requested a full tablet report, sending...
I20260505 20:18:52.453461 32286 ts_manager.cc:194] Registered new tserver with Master: c7ba04dcf34b4fb6ac6954b2dd83b63d (127.27.255.67:37321)
I20260505 20:18:52.453709 28669 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017775168s
I20260505 20:18:52.455381 32286 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60828
I20260505 20:18:52.504016 32286 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60854:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260505 20:18:52.547223 32527 tablet_service.cc:1511] Processing CreateTablet for tablet f8e80a7373444cce867276f34482a5bb (DEFAULT_TABLE table=test-workload [id=939ce59d11264a378133f98ae9b4f258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:52.548447 32527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8e80a7373444cce867276f34482a5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:52.551625 32453 tablet_service.cc:1511] Processing CreateTablet for tablet f8e80a7373444cce867276f34482a5bb (DEFAULT_TABLE table=test-workload [id=939ce59d11264a378133f98ae9b4f258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:52.552979 32453 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8e80a7373444cce867276f34482a5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:52.560281 32378 tablet_service.cc:1511] Processing CreateTablet for tablet f8e80a7373444cce867276f34482a5bb (DEFAULT_TABLE table=test-workload [id=939ce59d11264a378133f98ae9b4f258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:52.561365 32378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8e80a7373444cce867276f34482a5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:52.568264 32582 tablet_bootstrap.cc:492] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d: Bootstrap starting.
I20260505 20:18:52.570557 32583 tablet_bootstrap.cc:492] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Bootstrap starting.
I20260505 20:18:52.577144 32583 tablet_bootstrap.cc:654] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:52.577898 32584 tablet_bootstrap.cc:492] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58: Bootstrap starting.
I20260505 20:18:52.577144 32582 tablet_bootstrap.cc:654] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:52.583283 32583 tablet_bootstrap.cc:492] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: No bootstrap required, opened a new log
I20260505 20:18:52.583814 32583 ts_tablet_manager.cc:1403] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260505 20:18:52.584609 32582 tablet_bootstrap.cc:492] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d: No bootstrap required, opened a new log
I20260505 20:18:52.585075 32582 ts_tablet_manager.cc:1403] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20260505 20:18:52.585162 32584 tablet_bootstrap.cc:654] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:52.586612 32583 raft_consensus.cc:359] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } }
I20260505 20:18:52.587373 32583 raft_consensus.cc:385] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:52.587828 32583 raft_consensus.cc:740] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a9103fcca94cadb1b50424e37aca85, State: Initialized, Role: FOLLOWER
I20260505 20:18:52.588093 32582 raft_consensus.cc:359] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } }
I20260505 20:18:52.588928 32582 raft_consensus.cc:385] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:52.588614 32583 consensus_queue.cc:260] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } }
I20260505 20:18:52.589749 32582 raft_consensus.cc:740] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ba04dcf34b4fb6ac6954b2dd83b63d, State: Initialized, Role: FOLLOWER
I20260505 20:18:52.590296 32584 tablet_bootstrap.cc:492] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58: No bootstrap required, opened a new log
I20260505 20:18:52.590783 32584 ts_tablet_manager.cc:1403] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260505 20:18:52.590890 32582 consensus_queue.cc:260] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } }
I20260505 20:18:52.593605 32584 raft_consensus.cc:359] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } }
I20260505 20:18:52.594354 32584 raft_consensus.cc:385] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:52.594300 32562 heartbeater.cc:499] Master 127.27.255.126:39763 was elected leader, sending a full tablet report...
I20260505 20:18:52.594869 32584 raft_consensus.cc:740] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc702e3c0f2e456a96beeafb1754bb58, State: Initialized, Role: FOLLOWER
I20260505 20:18:52.595628 32584 consensus_queue.cc:260] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } }
I20260505 20:18:52.597878 32582 ts_tablet_manager.cc:1434] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260505 20:18:52.599797 32488 heartbeater.cc:499] Master 127.27.255.126:39763 was elected leader, sending a full tablet report...
I20260505 20:18:52.600351 32583 ts_tablet_manager.cc:1434] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
I20260505 20:18:52.602981 32413 heartbeater.cc:499] Master 127.27.255.126:39763 was elected leader, sending a full tablet report...
I20260505 20:18:52.603520 32584 ts_tablet_manager.cc:1434] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
I20260505 20:18:52.622012 32589 raft_consensus.cc:493] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:52.622543 32589 raft_consensus.cc:515] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } }
I20260505 20:18:52.624543 32589 leader_election.cc:290] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66:40909), cc702e3c0f2e456a96beeafb1754bb58 (127.27.255.65:40229)
I20260505 20:18:52.634822 32463 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8e80a7373444cce867276f34482a5bb" candidate_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a9103fcca94cadb1b50424e37aca85" is_pre_election: true
I20260505 20:18:52.635478 32463 raft_consensus.cc:2468] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7ba04dcf34b4fb6ac6954b2dd83b63d in term 0.
I20260505 20:18:52.635946 32388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8e80a7373444cce867276f34482a5bb" candidate_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc702e3c0f2e456a96beeafb1754bb58" is_pre_election: true
I20260505 20:18:52.636682 32388 raft_consensus.cc:2468] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7ba04dcf34b4fb6ac6954b2dd83b63d in term 0.
I20260505 20:18:52.636566 32501 leader_election.cc:304] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25a9103fcca94cadb1b50424e37aca85, c7ba04dcf34b4fb6ac6954b2dd83b63d; no voters:
I20260505 20:18:52.637881 32589 raft_consensus.cc:2804] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260505 20:18:52.638340 32589 raft_consensus.cc:493] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:52.639065 32589 raft_consensus.cc:3060] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:52.644279 32589 raft_consensus.cc:515] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } }
I20260505 20:18:52.645910 32589 leader_election.cc:290] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [CANDIDATE]: Term 1 election: Requested vote from peers 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66:40909), cc702e3c0f2e456a96beeafb1754bb58 (127.27.255.65:40229)
I20260505 20:18:52.646924 32388 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8e80a7373444cce867276f34482a5bb" candidate_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc702e3c0f2e456a96beeafb1754bb58"
I20260505 20:18:52.647022 32463 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8e80a7373444cce867276f34482a5bb" candidate_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a9103fcca94cadb1b50424e37aca85"
I20260505 20:18:52.647464 32388 raft_consensus.cc:3060] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:52.647550 32463 raft_consensus.cc:3060] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:52.652325 32463 raft_consensus.cc:2468] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7ba04dcf34b4fb6ac6954b2dd83b63d in term 1.
I20260505 20:18:52.652397 32388 raft_consensus.cc:2468] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7ba04dcf34b4fb6ac6954b2dd83b63d in term 1.
I20260505 20:18:52.653415 32501 leader_election.cc:304] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25a9103fcca94cadb1b50424e37aca85, c7ba04dcf34b4fb6ac6954b2dd83b63d; no voters:
I20260505 20:18:52.654187 32589 raft_consensus.cc:2804] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:52.655341 32589 raft_consensus.cc:697] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 1 LEADER]: Becoming Leader. State: Replica: c7ba04dcf34b4fb6ac6954b2dd83b63d, State: Running, Role: LEADER
I20260505 20:18:52.656128 32589 consensus_queue.cc:237] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } }
I20260505 20:18:52.664920 32285 catalog_manager.cc:5671] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d reported cstate change: term changed from 0 to 1, leader changed from <none> to c7ba04dcf34b4fb6ac6954b2dd83b63d (127.27.255.67). New cstate: current_term: 1 leader_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } } }
I20260505 20:18:52.954492 32453 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:52.956467 32378 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:52.957403 32527 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260505 20:18:53.064700 32589 consensus_queue.cc:1048] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260505 20:18:53.107683 32589 consensus_queue.cc:1048] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [LEADER]: Connected to new peer: Peer: permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
d3d2291c044040f4835225cf1659a9a9 | 127.27.255.126:39763 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
25a9103fcca94cadb1b50424e37aca85 | 127.27.255.66:40909 | HEALTHY | <none> | 0 | 0
c7ba04dcf34b4fb6ac6954b2dd83b63d | 127.27.255.67:37321 | HEALTHY | <none> | 1 | 0
cc702e3c0f2e456a96beeafb1754bb58 | 127.27.255.65:40229 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet f8e80a7373444cce867276f34482a5bb of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
c7ba04dcf34b4fb6ac6954b2dd83b63d (127.27.255.67:37321): RUNNING [LEADER]
25a9103fcca94cadb1b50424e37aca85 (127.27.255.66:40909): RUNNING
cc702e3c0f2e456a96beeafb1754bb58 (127.27.255.65:40229): RUNNING
All reported replicas are:
A = c7ba04dcf34b4fb6ac6954b2dd83b63d
B = 25a9103fcca94cadb1b50424e37aca85
C = cc702e3c0f2e456a96beeafb1754bb58
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260505 20:18:53.227329 28669 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260505 20:18:53.616709 32527 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260505 20:18:53.619622 32453 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:53.632012 32378 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
d3d2291c044040f4835225cf1659a9a9 | 127.27.255.126:39763 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
25a9103fcca94cadb1b50424e37aca85 | 127.27.255.66:40909 | HEALTHY | <none> | 0 | 0
c7ba04dcf34b4fb6ac6954b2dd83b63d | 127.27.255.67:37321 | HEALTHY | <none> | 1 | 0
cc702e3c0f2e456a96beeafb1754bb58 | 127.27.255.65:40229 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260505 20:18:53.913316 28669 log_verifier.cc:126] Checking tablet f8e80a7373444cce867276f34482a5bb
I20260505 20:18:53.991878 28669 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f8e80a7373444cce867276f34482a5bb
I20260505 20:18:53.992579 28669 tablet_replica.cc:333] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: stopping tablet replica
I20260505 20:18:53.993237 28669 raft_consensus.cc:2243] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:53.993920 28669 raft_consensus.cc:2272] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:54.259455 32453 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:54.261650 32527 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260505 20:18:54.272827 32378 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
d3d2291c044040f4835225cf1659a9a9 | 127.27.255.126:39763 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
25a9103fcca94cadb1b50424e37aca85 | 127.27.255.66:40909 | HEALTHY | <none> | 0 | 0
c7ba04dcf34b4fb6ac6954b2dd83b63d | 127.27.255.67:37321 | HEALTHY | <none> | 1 | 0
cc702e3c0f2e456a96beeafb1754bb58 | 127.27.255.65:40229 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet f8e80a7373444cce867276f34482a5bb of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c7ba04dcf34b4fb6ac6954b2dd83b63d (127.27.255.67:37321): RUNNING [LEADER]
25a9103fcca94cadb1b50424e37aca85 (127.27.255.66:40909): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
cc702e3c0f2e456a96beeafb1754bb58 (127.27.255.65:40229): RUNNING
All reported replicas are:
A = c7ba04dcf34b4fb6ac6954b2dd83b63d
B = 25a9103fcca94cadb1b50424e37aca85
C = cc702e3c0f2e456a96beeafb1754bb58
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | [config not available] | | |
C | A* B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260505 20:18:54.486057 28669 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260505 20:18:54.534094 32501 consensus_peers.cc:597] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d -> Peer 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66:40909): Couldn't send request to peer 25a9103fcca94cadb1b50424e37aca85. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260505 20:18:54.535457 32589 raft_consensus.cc:1012] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d: Attempting to remove follower 25a9103fcca94cadb1b50424e37aca85 from the Raft config. Reason: The tablet replica hosted on peer 25a9103fcca94cadb1b50424e37aca85 has failed
I20260505 20:18:54.537278 32589 consensus_queue.cc:237] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } }
I20260505 20:18:54.541956 32387 raft_consensus.cc:1275] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 1 FOLLOWER]: Refusing update from remote peer c7ba04dcf34b4fb6ac6954b2dd83b63d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260505 20:18:54.543241 32679 consensus_queue.cc:1048] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260505 20:18:54.550207 32589 raft_consensus.cc:2955] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } } }
I20260505 20:18:54.551703 32387 raft_consensus.cc:2955] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } } }
I20260505 20:18:54.563266 32285 catalog_manager.cc:5671] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d reported cstate change: config changed from index -1 to 2, VOTER 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66) evicted. New cstate: current_term: 1 leader_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } } }
I20260505 20:18:54.579818 32453 tablet_service.cc:1558] Processing DeleteTablet for tablet f8e80a7373444cce867276f34482a5bb with delete_type TABLET_DATA_TOMBSTONED (TS 25a9103fcca94cadb1b50424e37aca85 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:34804
I20260505 20:18:54.582036 32687 ts_tablet_manager.cc:1916] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260505 20:18:54.583002 32537 consensus_queue.cc:237] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } }
I20260505 20:18:54.587579 32387 raft_consensus.cc:1275] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 1 FOLLOWER]: Refusing update from remote peer c7ba04dcf34b4fb6ac6954b2dd83b63d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260505 20:18:54.588215 32501 consensus_peers.cc:597] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d -> Peer 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66:40909): Couldn't send request to peer 25a9103fcca94cadb1b50424e37aca85. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260505 20:18:54.588920 32589 consensus_queue.cc:1048] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260505 20:18:54.594590 32679 raft_consensus.cc:2955] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } }
I20260505 20:18:54.595464 32687 ts_tablet_manager.cc:1929] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260505 20:18:54.595883 32687 log.cc:1199] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Deleting WAL directory at /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1778012272869005-28669-0/minicluster-data/ts-1-root/wals/f8e80a7373444cce867276f34482a5bb
I20260505 20:18:54.596361 32387 raft_consensus.cc:2955] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } }
I20260505 20:18:54.597460 32271 catalog_manager.cc:5002] TS 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66:40909): tablet f8e80a7373444cce867276f34482a5bb (table test-workload [id=939ce59d11264a378133f98ae9b4f258]) successfully deleted
I20260505 20:18:54.604473 32272 catalog_manager.cc:5184] ChangeConfig:ADD_PEER:VOTER RPC for tablet f8e80a7373444cce867276f34482a5bb with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260505 20:18:54.607041 32285 catalog_manager.cc:5671] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d reported cstate change: config changed from index 2 to 3, VOTER 25a9103fcca94cadb1b50424e37aca85 (127.27.255.66) added. New cstate: current_term: 1 leader_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } } }
I20260505 20:18:54.870808 32453 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:54.879278 32527 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260505 20:18:54.885752 32378 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:55.014693 32714 ts_tablet_manager.cc:933] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Initiating tablet copy from peer c7ba04dcf34b4fb6ac6954b2dd83b63d (127.27.255.67:37321)
I20260505 20:18:55.026458 32714 tablet_copy_client.cc:323] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: tablet copy: Beginning tablet copy session from remote peer at address 127.27.255.67:37321
I20260505 20:18:55.049564 32547 tablet_copy_service.cc:140] P c7ba04dcf34b4fb6ac6954b2dd83b63d: Received BeginTabletCopySession request for tablet f8e80a7373444cce867276f34482a5bb from peer 25a9103fcca94cadb1b50424e37aca85 ({username='slave'} at 127.0.0.1:33718)
I20260505 20:18:55.050297 32547 tablet_copy_service.cc:161] P c7ba04dcf34b4fb6ac6954b2dd83b63d: Beginning new tablet copy session on tablet f8e80a7373444cce867276f34482a5bb from peer 25a9103fcca94cadb1b50424e37aca85 at {username='slave'} at 127.0.0.1:33718: session id = 25a9103fcca94cadb1b50424e37aca85-f8e80a7373444cce867276f34482a5bb
I20260505 20:18:55.056586 32547 tablet_copy_source_session.cc:215] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d: Tablet Copy: opened 0 blocks and 1 log segments
I20260505 20:18:55.059037 32714 ts_tablet_manager.cc:1916] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260505 20:18:55.071234 32714 ts_tablet_manager.cc:1929] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260505 20:18:55.071972 32714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8e80a7373444cce867276f34482a5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:55.078817 32714 tablet_copy_client.cc:806] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: tablet copy: Starting download of 0 data blocks...
I20260505 20:18:55.079365 32714 tablet_copy_client.cc:670] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: tablet copy: Starting download of 1 WAL segments...
I20260505 20:18:55.083865 32714 tablet_copy_client.cc:538] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260505 20:18:55.090992 32714 tablet_bootstrap.cc:492] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Bootstrap starting.
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
d3d2291c044040f4835225cf1659a9a9 | 127.27.255.126:39763 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
25a9103fcca94cadb1b50424e37aca85 | 127.27.255.66:40909 | HEALTHY | <none> | 0 | 0
c7ba04dcf34b4fb6ac6954b2dd83b63d | 127.27.255.67:37321 | HEALTHY | <none> | 1 | 0
cc702e3c0f2e456a96beeafb1754bb58 | 127.27.255.65:40229 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet f8e80a7373444cce867276f34482a5bb of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c7ba04dcf34b4fb6ac6954b2dd83b63d (127.27.255.67:37321): RUNNING [LEADER]
cc702e3c0f2e456a96beeafb1754bb58 (127.27.255.65:40229): RUNNING
25a9103fcca94cadb1b50424e37aca85 (127.27.255.66:40909): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = c7ba04dcf34b4fb6ac6954b2dd83b63d
B = cc702e3c0f2e456a96beeafb1754bb58
C = 25a9103fcca94cadb1b50424e37aca85
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260505 20:18:55.110667 28669 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260505 20:18:55.117673 32714 tablet_bootstrap.cc:492] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 20:18:55.118428 32714 tablet_bootstrap.cc:492] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Bootstrap complete.
I20260505 20:18:55.119019 32714 ts_tablet_manager.cc:1403] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.012s
I20260505 20:18:55.121346 32714 raft_consensus.cc:359] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } }
I20260505 20:18:55.122007 32714 raft_consensus.cc:740] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a9103fcca94cadb1b50424e37aca85, State: Initialized, Role: FOLLOWER
I20260505 20:18:55.122617 32714 consensus_queue.cc:260] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c7ba04dcf34b4fb6ac6954b2dd83b63d" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 37321 } } peers { permanent_uuid: "cc702e3c0f2e456a96beeafb1754bb58" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 40229 } } peers { permanent_uuid: "25a9103fcca94cadb1b50424e37aca85" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 40909 } }
I20260505 20:18:55.124742 32714 ts_tablet_manager.cc:1434] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260505 20:18:55.126618 32547 tablet_copy_service.cc:342] P c7ba04dcf34b4fb6ac6954b2dd83b63d: Request end of tablet copy session 25a9103fcca94cadb1b50424e37aca85-f8e80a7373444cce867276f34482a5bb received from {username='slave'} at 127.0.0.1:33718
I20260505 20:18:55.127048 32547 tablet_copy_service.cc:434] P c7ba04dcf34b4fb6ac6954b2dd83b63d: ending tablet copy session 25a9103fcca94cadb1b50424e37aca85-f8e80a7373444cce867276f34482a5bb on tablet f8e80a7373444cce867276f34482a5bb with peer 25a9103fcca94cadb1b50424e37aca85
I20260505 20:18:55.574330 32453 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:55.580412 32527 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260505 20:18:55.583699 32378 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:55.613682 32463 raft_consensus.cc:1217] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
d3d2291c044040f4835225cf1659a9a9 | 127.27.255.126:39763 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
25a9103fcca94cadb1b50424e37aca85 | 127.27.255.66:40909 | HEALTHY | <none> | 0 | 0
c7ba04dcf34b4fb6ac6954b2dd83b63d | 127.27.255.67:37321 | HEALTHY | <none> | 1 | 0
cc702e3c0f2e456a96beeafb1754bb58 | 127.27.255.65:40229 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet f8e80a7373444cce867276f34482a5bb of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
c7ba04dcf34b4fb6ac6954b2dd83b63d (127.27.255.67:37321): RUNNING [LEADER]
cc702e3c0f2e456a96beeafb1754bb58 (127.27.255.65:40229): RUNNING
25a9103fcca94cadb1b50424e37aca85 (127.27.255.66:40909): RUNNING
All reported replicas are:
A = c7ba04dcf34b4fb6ac6954b2dd83b63d
B = cc702e3c0f2e456a96beeafb1754bb58
C = 25a9103fcca94cadb1b50424e37aca85
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | A B C | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260505 20:18:55.799925 28669 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260505 20:18:56.399176 32527 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260505 20:18:56.401211 32453 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:56.402868 32378 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
d3d2291c044040f4835225cf1659a9a9 | 127.27.255.126:39763 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
25a9103fcca94cadb1b50424e37aca85 | 127.27.255.66:40909 | HEALTHY | <none> | 0 | 0
c7ba04dcf34b4fb6ac6954b2dd83b63d | 127.27.255.67:37321 | HEALTHY | <none> | 1 | 0
cc702e3c0f2e456a96beeafb1754bb58 | 127.27.255.65:40229 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260505 20:18:56.615857 28669 log_verifier.cc:126] Checking tablet f8e80a7373444cce867276f34482a5bb
I20260505 20:18:56.691642 28669 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f8e80a7373444cce867276f34482a5bb
I20260505 20:18:56.695577 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:18:56.730808 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:56.731904 28669 tablet_replica.cc:333] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58: stopping tablet replica
I20260505 20:18:56.732492 28669 raft_consensus.cc:2243] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:56.733182 28669 raft_consensus.cc:2272] T f8e80a7373444cce867276f34482a5bb P cc702e3c0f2e456a96beeafb1754bb58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:56.756949 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:18:56.770921 28669 tablet_server.cc:179] TabletServer@127.27.255.66:0 shutting down...
I20260505 20:18:56.805395 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:56.806282 28669 tablet_replica.cc:333] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85: stopping tablet replica
I20260505 20:18:56.807176 28669 raft_consensus.cc:2243] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260505 20:18:56.807680 28669 raft_consensus.cc:2272] T f8e80a7373444cce867276f34482a5bb P 25a9103fcca94cadb1b50424e37aca85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:56.830973 28669 tablet_server.cc:196] TabletServer@127.27.255.66:0 shutdown complete.
I20260505 20:18:56.843762 28669 tablet_server.cc:179] TabletServer@127.27.255.67:0 shutting down...
I20260505 20:18:56.872370 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:18:56.873276 28669 tablet_replica.cc:333] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d: stopping tablet replica
I20260505 20:18:56.873980 28669 raft_consensus.cc:2243] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:56.874950 28669 raft_consensus.cc:2272] T f8e80a7373444cce867276f34482a5bb P c7ba04dcf34b4fb6ac6954b2dd83b63d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:56.898407 28669 tablet_server.cc:196] TabletServer@127.27.255.67:0 shutdown complete.
I20260505 20:18:56.911911 28669 master.cc:562] Master@127.27.255.126:39763 shutting down...
I20260505 20:18:56.936901 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 2 LEADER]: Raft consensus shutting down.
I20260505 20:18:56.937685 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:56.938084 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P d3d2291c044040f4835225cf1659a9a9: stopping tablet replica
I20260505 20:18:56.958791 28669 master.cc:584] Master@127.27.255.126:39763 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (5125 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20260505 20:18:56.995522 28669 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.255.126:39413
I20260505 20:18:56.996893 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:57.003484 309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:57.004469 310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:57.006093 312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:57.006377 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:57.007524 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:57.007710 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:57.008163 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012337008098 us; error 0 us; skew 500 ppm
I20260505 20:18:57.009047 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:57.012120 28669 webserver.cc:492] Webserver started at http://127.27.255.126:42787/ using document root <none> and password file <none>
I20260505 20:18:57.012672 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:57.012841 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:57.013186 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:57.014580 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/master-0-root/instance:
uuid: "c692829e03fc452ca8ff8ec11129cc61"
format_stamp: "Formatted at 2026-05-05 20:18:57 on dist-test-slave-dnnj"
I20260505 20:18:57.019295 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260505 20:18:57.023067 317 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.024065 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260505 20:18:57.024374 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/master-0-root
uuid: "c692829e03fc452ca8ff8ec11129cc61"
format_stamp: "Formatted at 2026-05-05 20:18:57 on dist-test-slave-dnnj"
I20260505 20:18:57.024647 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:57.040926 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:57.042294 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:57.084153 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.126:39413
I20260505 20:18:57.084249 368 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.126:39413 every 8 connection(s)
I20260505 20:18:57.088269 369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:57.099223 369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61: Bootstrap starting.
I20260505 20:18:57.104221 369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:57.108809 369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61: No bootstrap required, opened a new log
I20260505 20:18:57.111135 369 raft_consensus.cc:359] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c692829e03fc452ca8ff8ec11129cc61" member_type: VOTER }
I20260505 20:18:57.111529 369 raft_consensus.cc:385] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:57.111822 369 raft_consensus.cc:740] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c692829e03fc452ca8ff8ec11129cc61, State: Initialized, Role: FOLLOWER
I20260505 20:18:57.112425 369 consensus_queue.cc:260] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c692829e03fc452ca8ff8ec11129cc61" member_type: VOTER }
I20260505 20:18:57.112890 369 raft_consensus.cc:399] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 20:18:57.113094 369 raft_consensus.cc:493] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 20:18:57.113384 369 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:57.118234 369 raft_consensus.cc:515] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c692829e03fc452ca8ff8ec11129cc61" member_type: VOTER }
I20260505 20:18:57.118952 369 leader_election.cc:304] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c692829e03fc452ca8ff8ec11129cc61; no voters:
I20260505 20:18:57.120127 369 leader_election.cc:290] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 20:18:57.120623 372 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:57.122071 372 raft_consensus.cc:697] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 1 LEADER]: Becoming Leader. State: Replica: c692829e03fc452ca8ff8ec11129cc61, State: Running, Role: LEADER
I20260505 20:18:57.122856 372 consensus_queue.cc:237] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c692829e03fc452ca8ff8ec11129cc61" member_type: VOTER }
I20260505 20:18:57.123241 369 sys_catalog.cc:565] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 20:18:57.126276 374 sys_catalog.cc:455] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c692829e03fc452ca8ff8ec11129cc61. Latest consensus state: current_term: 1 leader_uuid: "c692829e03fc452ca8ff8ec11129cc61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c692829e03fc452ca8ff8ec11129cc61" member_type: VOTER } }
I20260505 20:18:57.127044 374 sys_catalog.cc:458] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:57.126426 373 sys_catalog.cc:455] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c692829e03fc452ca8ff8ec11129cc61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c692829e03fc452ca8ff8ec11129cc61" member_type: VOTER } }
I20260505 20:18:57.128346 373 sys_catalog.cc:458] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [sys.catalog]: This master's current role is: LEADER
I20260505 20:18:57.131503 378 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 20:18:57.139714 378 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 20:18:57.140714 28669 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260505 20:18:57.148842 378 catalog_manager.cc:1357] Generated new cluster ID: b3972dbc212a477694a387c8e6660b72
I20260505 20:18:57.149161 378 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 20:18:57.163838 378 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 20:18:57.165171 378 catalog_manager.cc:1514] Loading token signing keys...
I20260505 20:18:57.183099 378 catalog_manager.cc:6044] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61: Generated new TSK 0
I20260505 20:18:57.183718 378 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 20:18:57.208244 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:57.214762 390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:57.216435 391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:57.217844 393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:57.219398 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:57.220396 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:57.220595 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:57.220782 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012337220768 us; error 0 us; skew 500 ppm
I20260505 20:18:57.221325 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:57.223922 28669 webserver.cc:492] Webserver started at http://127.27.255.65:46081/ using document root <none> and password file <none>
I20260505 20:18:57.224403 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:57.224604 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:57.224959 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:57.226089 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-0-root/instance:
uuid: "36de6d0f384b45f995635dfb983c6fa7"
format_stamp: "Formatted at 2026-05-05 20:18:57 on dist-test-slave-dnnj"
I20260505 20:18:57.230643 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260505 20:18:57.234110 398 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.235208 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260505 20:18:57.235543 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-0-root
uuid: "36de6d0f384b45f995635dfb983c6fa7"
format_stamp: "Formatted at 2026-05-05 20:18:57 on dist-test-slave-dnnj"
I20260505 20:18:57.235818 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:57.250249 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:57.251330 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:57.252807 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:57.255219 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:57.255399 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.255579 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:57.255714 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.303032 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.65:46287
I20260505 20:18:57.303126 460 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.65:46287 every 8 connection(s)
I20260505 20:18:57.308255 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:57.315528 465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:57.315984 466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:57.319379 468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:57.320339 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:57.321149 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:57.321332 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:57.321477 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012337321459 us; error 0 us; skew 500 ppm
I20260505 20:18:57.322788 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:57.326013 28669 webserver.cc:492] Webserver started at http://127.27.255.66:41921/ using document root <none> and password file <none>
I20260505 20:18:57.326591 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:57.326826 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:57.327157 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:57.327378 461 heartbeater.cc:344] Connected to a master server at 127.27.255.126:39413
I20260505 20:18:57.328004 461 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:57.328310 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-1-root/instance:
uuid: "2d71a29bf9bc444fa630af946677a4c3"
format_stamp: "Formatted at 2026-05-05 20:18:57 on dist-test-slave-dnnj"
I20260505 20:18:57.329095 461 heartbeater.cc:507] Master 127.27.255.126:39413 requested a full tablet report, sending...
I20260505 20:18:57.331280 334 ts_manager.cc:194] Registered new tserver with Master: 36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65:46287)
I20260505 20:18:57.333688 334 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33604
I20260505 20:18:57.334122 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260505 20:18:57.337992 473 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.338884 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260505 20:18:57.339171 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-1-root
uuid: "2d71a29bf9bc444fa630af946677a4c3"
format_stamp: "Formatted at 2026-05-05 20:18:57 on dist-test-slave-dnnj"
I20260505 20:18:57.339422 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:57.352537 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:57.353721 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:57.355134 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:57.357645 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:57.357885 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.358144 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:57.358318 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.406589 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.66:45409
I20260505 20:18:57.406687 535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.66:45409 every 8 connection(s)
I20260505 20:18:57.411830 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:57.419098 539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:57.420033 540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:57.423643 28669 server_base.cc:1061] running on GCE node
W20260505 20:18:57.423888 542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:57.424134 536 heartbeater.cc:344] Connected to a master server at 127.27.255.126:39413
I20260505 20:18:57.424633 536 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:57.424959 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:57.425228 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:57.425366 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012337425354 us; error 0 us; skew 500 ppm
I20260505 20:18:57.425390 536 heartbeater.cc:507] Master 127.27.255.126:39413 requested a full tablet report, sending...
I20260505 20:18:57.425972 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:57.427866 334 ts_manager.cc:194] Registered new tserver with Master: 2d71a29bf9bc444fa630af946677a4c3 (127.27.255.66:45409)
I20260505 20:18:57.428905 28669 webserver.cc:492] Webserver started at http://127.27.255.67:40725/ using document root <none> and password file <none>
I20260505 20:18:57.429406 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:57.429675 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:57.429574 334 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33612
I20260505 20:18:57.430079 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:57.431502 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-2-root/instance:
uuid: "a5b2e66d4e6645daaa190ab40f2def19"
format_stamp: "Formatted at 2026-05-05 20:18:57 on dist-test-slave-dnnj"
I20260505 20:18:57.436374 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260505 20:18:57.440029 547 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.440977 28669 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260505 20:18:57.441316 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-2-root
uuid: "a5b2e66d4e6645daaa190ab40f2def19"
format_stamp: "Formatted at 2026-05-05 20:18:57 on dist-test-slave-dnnj"
I20260505 20:18:57.441706 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:57.456318 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:57.457646 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:57.459332 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:57.461668 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:57.461908 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.462169 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:57.462345 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.513885 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.67:36591
I20260505 20:18:57.514195 609 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.67:36591 every 8 connection(s)
I20260505 20:18:57.520272 28669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 20:18:57.529834 614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:57.530655 613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 20:18:57.534551 616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 20:18:57.535260 28669 server_base.cc:1061] running on GCE node
I20260505 20:18:57.536146 610 heartbeater.cc:344] Connected to a master server at 127.27.255.126:39413
I20260505 20:18:57.536152 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260505 20:18:57.536887 28669 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260505 20:18:57.537014 610 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:57.537132 28669 hybrid_clock.cc:648] HybridClock initialized: now 1778012337537113 us; error 0 us; skew 500 ppm
I20260505 20:18:57.538283 28669 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 20:18:57.538352 610 heartbeater.cc:507] Master 127.27.255.126:39413 requested a full tablet report, sending...
I20260505 20:18:57.541725 28669 webserver.cc:492] Webserver started at http://127.27.255.68:45905/ using document root <none> and password file <none>
I20260505 20:18:57.541728 334 ts_manager.cc:194] Registered new tserver with Master: a5b2e66d4e6645daaa190ab40f2def19 (127.27.255.67:36591)
I20260505 20:18:57.542410 28669 fs_manager.cc:362] Metadata directory not provided
I20260505 20:18:57.542665 28669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 20:18:57.542939 28669 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 20:18:57.543781 334 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33624
I20260505 20:18:57.544301 28669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-3-root/instance:
uuid: "b4b867020b7d41db8d83c3a52ef5a127"
format_stamp: "Formatted at 2026-05-05 20:18:57 on dist-test-slave-dnnj"
I20260505 20:18:57.549345 28669 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260505 20:18:57.554059 621 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.555011 28669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260505 20:18:57.555330 28669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-3-root
uuid: "b4b867020b7d41db8d83c3a52ef5a127"
format_stamp: "Formatted at 2026-05-05 20:18:57 on dist-test-slave-dnnj"
I20260505 20:18:57.555616 28669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 20:18:57.579694 28669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 20:18:57.581074 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 20:18:57.582726 28669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 20:18:57.585059 28669 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 20:18:57.585246 28669 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.585551 28669 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 20:18:57.585716 28669 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 20:18:57.630352 28669 rpc_server.cc:307] RPC server started. Bound to: 127.27.255.68:37635
I20260505 20:18:57.630481 683 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.255.68:37635 every 8 connection(s)
I20260505 20:18:57.644423 684 heartbeater.cc:344] Connected to a master server at 127.27.255.126:39413
I20260505 20:18:57.644825 684 heartbeater.cc:461] Registering TS with master...
I20260505 20:18:57.645663 684 heartbeater.cc:507] Master 127.27.255.126:39413 requested a full tablet report, sending...
I20260505 20:18:57.648168 334 ts_manager.cc:194] Registered new tserver with Master: b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635)
I20260505 20:18:57.649274 28669 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.015882458s
I20260505 20:18:57.649845 334 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33630
I20260505 20:18:57.698787 334 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33646:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260505 20:18:57.750041 426 tablet_service.cc:1511] Processing CreateTablet for tablet 71bd71769aa04036ab6a88793b08e0ca (DEFAULT_TABLE table=test-workload [id=280a291591584e8387725db070ed5a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:57.751255 426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71bd71769aa04036ab6a88793b08e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:57.751107 575 tablet_service.cc:1511] Processing CreateTablet for tablet 71bd71769aa04036ab6a88793b08e0ca (DEFAULT_TABLE table=test-workload [id=280a291591584e8387725db070ed5a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:57.752543 575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71bd71769aa04036ab6a88793b08e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:57.761056 649 tablet_service.cc:1511] Processing CreateTablet for tablet 71bd71769aa04036ab6a88793b08e0ca (DEFAULT_TABLE table=test-workload [id=280a291591584e8387725db070ed5a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 20:18:57.762265 649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71bd71769aa04036ab6a88793b08e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:18:57.770466 707 tablet_bootstrap.cc:492] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7: Bootstrap starting.
I20260505 20:18:57.773921 708 tablet_bootstrap.cc:492] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19: Bootstrap starting.
I20260505 20:18:57.779542 709 tablet_bootstrap.cc:492] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127: Bootstrap starting.
I20260505 20:18:57.778427 707 tablet_bootstrap.cc:654] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:57.780316 708 tablet_bootstrap.cc:654] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:57.785657 707 tablet_bootstrap.cc:492] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7: No bootstrap required, opened a new log
I20260505 20:18:57.786103 707 ts_tablet_manager.cc:1403] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.002s
I20260505 20:18:57.786461 708 tablet_bootstrap.cc:492] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19: No bootstrap required, opened a new log
I20260505 20:18:57.786959 708 ts_tablet_manager.cc:1403] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260505 20:18:57.787204 709 tablet_bootstrap.cc:654] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127: Neither blocks nor log segments found. Creating new log.
I20260505 20:18:57.788492 707 raft_consensus.cc:359] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:18:57.789117 707 raft_consensus.cc:385] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:57.789386 707 raft_consensus.cc:740] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36de6d0f384b45f995635dfb983c6fa7, State: Initialized, Role: FOLLOWER
I20260505 20:18:57.790140 708 raft_consensus.cc:359] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:18:57.791374 708 raft_consensus.cc:385] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:57.790123 707 consensus_queue.cc:260] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:18:57.791695 708 raft_consensus.cc:740] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b2e66d4e6645daaa190ab40f2def19, State: Initialized, Role: FOLLOWER
I20260505 20:18:57.793115 708 consensus_queue.cc:260] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:18:57.793980 709 tablet_bootstrap.cc:492] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127: No bootstrap required, opened a new log
I20260505 20:18:57.794760 709 ts_tablet_manager.cc:1403] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260505 20:18:57.795974 610 heartbeater.cc:499] Master 127.27.255.126:39413 was elected leader, sending a full tablet report...
I20260505 20:18:57.796296 708 ts_tablet_manager.cc:1434] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260505 20:18:57.797709 461 heartbeater.cc:499] Master 127.27.255.126:39413 was elected leader, sending a full tablet report...
I20260505 20:18:57.798244 709 raft_consensus.cc:359] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:18:57.798988 709 raft_consensus.cc:385] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 20:18:57.799273 709 raft_consensus.cc:740] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4b867020b7d41db8d83c3a52ef5a127, State: Initialized, Role: FOLLOWER
I20260505 20:18:57.799612 707 ts_tablet_manager.cc:1434] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7: Time spent starting tablet: real 0.013s user 0.006s sys 0.004s
I20260505 20:18:57.800141 709 consensus_queue.cc:260] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:18:57.802951 684 heartbeater.cc:499] Master 127.27.255.126:39413 was elected leader, sending a full tablet report...
I20260505 20:18:57.803220 709 ts_tablet_manager.cc:1434] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260505 20:18:57.927007 715 raft_consensus.cc:493] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 20:18:57.927737 715 raft_consensus.cc:515] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:18:57.929894 715 leader_election.cc:290] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b2e66d4e6645daaa190ab40f2def19 (127.27.255.67:36591), 36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65:46287)
I20260505 20:18:57.940267 436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71bd71769aa04036ab6a88793b08e0ca" candidate_uuid: "b4b867020b7d41db8d83c3a52ef5a127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36de6d0f384b45f995635dfb983c6fa7" is_pre_election: true
I20260505 20:18:57.940866 585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71bd71769aa04036ab6a88793b08e0ca" candidate_uuid: "b4b867020b7d41db8d83c3a52ef5a127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b2e66d4e6645daaa190ab40f2def19" is_pre_election: true
I20260505 20:18:57.941601 585 raft_consensus.cc:2468] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4b867020b7d41db8d83c3a52ef5a127 in term 0.
I20260505 20:18:57.941109 436 raft_consensus.cc:2468] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4b867020b7d41db8d83c3a52ef5a127 in term 0.
I20260505 20:18:57.942785 623 leader_election.cc:304] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b2e66d4e6645daaa190ab40f2def19, b4b867020b7d41db8d83c3a52ef5a127; no voters:
I20260505 20:18:57.943810 715 raft_consensus.cc:2804] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260505 20:18:57.944154 715 raft_consensus.cc:493] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 20:18:57.944494 715 raft_consensus.cc:3060] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:57.949560 715 raft_consensus.cc:515] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:18:57.951365 715 leader_election.cc:290] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [CANDIDATE]: Term 1 election: Requested vote from peers a5b2e66d4e6645daaa190ab40f2def19 (127.27.255.67:36591), 36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65:46287)
I20260505 20:18:57.952296 585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71bd71769aa04036ab6a88793b08e0ca" candidate_uuid: "b4b867020b7d41db8d83c3a52ef5a127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b2e66d4e6645daaa190ab40f2def19"
I20260505 20:18:57.952356 436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71bd71769aa04036ab6a88793b08e0ca" candidate_uuid: "b4b867020b7d41db8d83c3a52ef5a127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36de6d0f384b45f995635dfb983c6fa7"
I20260505 20:18:57.953058 585 raft_consensus.cc:3060] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:57.953224 436 raft_consensus.cc:3060] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260505 20:18:57.958793 585 raft_consensus.cc:2468] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4b867020b7d41db8d83c3a52ef5a127 in term 1.
I20260505 20:18:57.958859 436 raft_consensus.cc:2468] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4b867020b7d41db8d83c3a52ef5a127 in term 1.
I20260505 20:18:57.960078 623 leader_election.cc:304] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b2e66d4e6645daaa190ab40f2def19, b4b867020b7d41db8d83c3a52ef5a127; no voters:
I20260505 20:18:57.960887 715 raft_consensus.cc:2804] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 20:18:57.961855 715 raft_consensus.cc:697] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 1 LEADER]: Becoming Leader. State: Replica: b4b867020b7d41db8d83c3a52ef5a127, State: Running, Role: LEADER
I20260505 20:18:57.962512 715 consensus_queue.cc:237] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:18:57.969472 332 catalog_manager.cc:5671] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68). New cstate: current_term: 1 leader_uuid: "b4b867020b7d41db8d83c3a52ef5a127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } health_report { overall_health: HEALTHY } } }
I20260505 20:18:58.271514 501 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:58.271940 426 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:58.275197 575 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:58.281100 649 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260505 20:18:58.415900 715 consensus_queue.cc:1048] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260505 20:18:58.432739 536 heartbeater.cc:499] Master 127.27.255.126:39413 was elected leader, sending a full tablet report...
I20260505 20:18:58.504273 715 consensus_queue.cc:1048] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c692829e03fc452ca8ff8ec11129cc61 | 127.27.255.126:39413 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | HEALTHY | <none> | 0 | 0
36de6d0f384b45f995635dfb983c6fa7 | 127.27.255.65:46287 | HEALTHY | <none> | 0 | 0
a5b2e66d4e6645daaa190ab40f2def19 | 127.27.255.67:36591 | HEALTHY | <none> | 0 | 0
b4b867020b7d41db8d83c3a52ef5a127 | 127.27.255.68:37635 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 71bd71769aa04036ab6a88793b08e0ca of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
a5b2e66d4e6645daaa190ab40f2def19 (127.27.255.67:36591): RUNNING
36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65:46287): RUNNING
b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635): RUNNING [LEADER]
All reported replicas are:
A = a5b2e66d4e6645daaa190ab40f2def19
B = 36de6d0f384b45f995635dfb983c6fa7
C = b4b867020b7d41db8d83c3a52ef5a127
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260505 20:18:58.628765 28669 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260505 20:18:59.057642 501 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:59.063664 426 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:59.064321 575 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:59.065245 649 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c692829e03fc452ca8ff8ec11129cc61 | 127.27.255.126:39413 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | HEALTHY | <none> | 0 | 0
36de6d0f384b45f995635dfb983c6fa7 | 127.27.255.65:46287 | HEALTHY | <none> | 0 | 0
a5b2e66d4e6645daaa190ab40f2def19 | 127.27.255.67:36591 | HEALTHY | <none> | 0 | 0
b4b867020b7d41db8d83c3a52ef5a127 | 127.27.255.68:37635 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260505 20:18:59.336583 28669 log_verifier.cc:126] Checking tablet 71bd71769aa04036ab6a88793b08e0ca
I20260505 20:18:59.413955 28669 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 71bd71769aa04036ab6a88793b08e0ca
I20260505 20:18:59.414429 28669 tablet_replica.cc:333] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127: stopping tablet replica
I20260505 20:18:59.414892 28669 raft_consensus.cc:2243] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:18:59.415689 28669 raft_consensus.cc:2272] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:18:59.674201 501 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:59.685348 426 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:59.688349 575 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:18:59.691604 649 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c692829e03fc452ca8ff8ec11129cc61 | 127.27.255.126:39413 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | HEALTHY | <none> | 0 | 0
36de6d0f384b45f995635dfb983c6fa7 | 127.27.255.65:46287 | HEALTHY | <none> | 0 | 0
a5b2e66d4e6645daaa190ab40f2def19 | 127.27.255.67:36591 | HEALTHY | <none> | 0 | 0
b4b867020b7d41db8d83c3a52ef5a127 | 127.27.255.68:37635 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 71bd71769aa04036ab6a88793b08e0ca of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
a5b2e66d4e6645daaa190ab40f2def19 (127.27.255.67:36591): RUNNING
36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65:46287): RUNNING
b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = a5b2e66d4e6645daaa190ab40f2def19
B = 36de6d0f384b45f995635dfb983c6fa7
C = b4b867020b7d41db8d83c3a52ef5a127
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260505 20:18:59.935807 28669 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260505 20:19:00.360368 501 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:00.360476 575 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:00.362421 426 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:00.365324 649 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:00.591558 847 raft_consensus.cc:493] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4b867020b7d41db8d83c3a52ef5a127)
I20260505 20:19:00.592327 847 raft_consensus.cc:515] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:19:00.596096 847 leader_election.cc:290] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65:46287), b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635)
I20260505 20:19:00.645797 436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71bd71769aa04036ab6a88793b08e0ca" candidate_uuid: "a5b2e66d4e6645daaa190ab40f2def19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "36de6d0f384b45f995635dfb983c6fa7" is_pre_election: true
I20260505 20:19:00.652582 659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71bd71769aa04036ab6a88793b08e0ca" candidate_uuid: "a5b2e66d4e6645daaa190ab40f2def19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b4b867020b7d41db8d83c3a52ef5a127" is_pre_election: true
W20260505 20:19:00.654178 548 leader_election.cc:343] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635): Illegal state: cannot vote while shut down
I20260505 20:19:00.654690 548 leader_election.cc:304] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5b2e66d4e6645daaa190ab40f2def19; no voters: 36de6d0f384b45f995635dfb983c6fa7, b4b867020b7d41db8d83c3a52ef5a127
I20260505 20:19:00.655586 847 raft_consensus.cc:2749] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c692829e03fc452ca8ff8ec11129cc61 | 127.27.255.126:39413 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | HEALTHY | <none> | 0 | 0
36de6d0f384b45f995635dfb983c6fa7 | 127.27.255.65:46287 | HEALTHY | <none> | 0 | 0
a5b2e66d4e6645daaa190ab40f2def19 | 127.27.255.67:36591 | HEALTHY | <none> | 0 | 0
b4b867020b7d41db8d83c3a52ef5a127 | 127.27.255.68:37635 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 71bd71769aa04036ab6a88793b08e0ca of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
a5b2e66d4e6645daaa190ab40f2def19 (127.27.255.67:36591): RUNNING
36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65:46287): RUNNING
b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = a5b2e66d4e6645daaa190ab40f2def19
B = 36de6d0f384b45f995635dfb983c6fa7
C = b4b867020b7d41db8d83c3a52ef5a127
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260505 20:19:00.694314 28669 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260505 20:19:00.877830 850 raft_consensus.cc:493] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b4b867020b7d41db8d83c3a52ef5a127)
I20260505 20:19:00.878568 850 raft_consensus.cc:515] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:19:00.881254 850 leader_election.cc:290] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5b2e66d4e6645daaa190ab40f2def19 (127.27.255.67:36591), b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635)
I20260505 20:19:00.895994 585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71bd71769aa04036ab6a88793b08e0ca" candidate_uuid: "36de6d0f384b45f995635dfb983c6fa7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5b2e66d4e6645daaa190ab40f2def19" is_pre_election: true
I20260505 20:19:00.895994 659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71bd71769aa04036ab6a88793b08e0ca" candidate_uuid: "36de6d0f384b45f995635dfb983c6fa7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b4b867020b7d41db8d83c3a52ef5a127" is_pre_election: true
I20260505 20:19:00.897301 585 raft_consensus.cc:2468] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36de6d0f384b45f995635dfb983c6fa7 in term 1.
I20260505 20:19:00.898530 400 leader_election.cc:304] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36de6d0f384b45f995635dfb983c6fa7, a5b2e66d4e6645daaa190ab40f2def19; no voters:
W20260505 20:19:00.899217 399 leader_election.cc:343] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635): Illegal state: cannot vote while shut down
I20260505 20:19:00.899418 850 raft_consensus.cc:2804] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260505 20:19:00.900148 850 raft_consensus.cc:493] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b4b867020b7d41db8d83c3a52ef5a127)
I20260505 20:19:00.900524 850 raft_consensus.cc:3060] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 1 FOLLOWER]: Advancing to term 2
I20260505 20:19:00.908226 850 raft_consensus.cc:515] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:19:00.910400 850 leader_election.cc:290] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [CANDIDATE]: Term 2 election: Requested vote from peers a5b2e66d4e6645daaa190ab40f2def19 (127.27.255.67:36591), b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635)
I20260505 20:19:00.911830 585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71bd71769aa04036ab6a88793b08e0ca" candidate_uuid: "36de6d0f384b45f995635dfb983c6fa7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a5b2e66d4e6645daaa190ab40f2def19"
I20260505 20:19:00.911914 659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71bd71769aa04036ab6a88793b08e0ca" candidate_uuid: "36de6d0f384b45f995635dfb983c6fa7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b4b867020b7d41db8d83c3a52ef5a127"
I20260505 20:19:00.912909 585 raft_consensus.cc:3060] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 1 FOLLOWER]: Advancing to term 2
W20260505 20:19:00.914119 399 leader_election.cc:343] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635): Illegal state: cannot vote while shut down
I20260505 20:19:00.920163 585 raft_consensus.cc:2468] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36de6d0f384b45f995635dfb983c6fa7 in term 2.
I20260505 20:19:00.921788 400 leader_election.cc:304] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36de6d0f384b45f995635dfb983c6fa7, a5b2e66d4e6645daaa190ab40f2def19; no voters: b4b867020b7d41db8d83c3a52ef5a127
I20260505 20:19:00.922660 850 raft_consensus.cc:2804] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 2 FOLLOWER]: Leader election won for term 2
I20260505 20:19:00.924476 850 raft_consensus.cc:697] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 2 LEADER]: Becoming Leader. State: Replica: 36de6d0f384b45f995635dfb983c6fa7, State: Running, Role: LEADER
I20260505 20:19:00.926169 850 consensus_queue.cc:237] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } }
I20260505 20:19:00.936947 331 catalog_manager.cc:5671] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 reported cstate change: term changed from 1 to 2, leader changed from b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68) to 36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65). New cstate: current_term: 2 leader_uuid: "36de6d0f384b45f995635dfb983c6fa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4b867020b7d41db8d83c3a52ef5a127" member_type: VOTER last_known_addr { host: "127.27.255.68" port: 37635 } health_report { overall_health: UNKNOWN } } }
I20260505 20:19:01.224268 501 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:01.224702 426 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260505 20:19:01.230005 575 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:01.233321 649 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:01.371470 399 consensus_queue.cc:579] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [LEADER]: The tablet replica hosted on peer b4b867020b7d41db8d83c3a52ef5a127 has failed
W20260505 20:19:01.372550 399 consensus_peers.cc:597] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 -> Peer b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635): Couldn't send request to peer b4b867020b7d41db8d83c3a52ef5a127. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260505 20:19:01.408876 585 raft_consensus.cc:1275] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 2 FOLLOWER]: Refusing update from remote peer 36de6d0f384b45f995635dfb983c6fa7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260505 20:19:01.411868 856 consensus_queue.cc:1048] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260505 20:19:01.503154 436 consensus_queue.cc:237] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } }
I20260505 20:19:01.509958 585 raft_consensus.cc:1275] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 2 FOLLOWER]: Refusing update from remote peer 36de6d0f384b45f995635dfb983c6fa7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260505 20:19:01.511673 856 consensus_queue.cc:1048] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260505 20:19:01.519835 850 raft_consensus.cc:2955] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } }
I20260505 20:19:01.523581 585 raft_consensus.cc:2955] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } }
I20260505 20:19:01.535075 321 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 71bd71769aa04036ab6a88793b08e0ca with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260505 20:19:01.541798 331 catalog_manager.cc:5671] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 reported cstate change: config changed from index -1 to 3, VOTER b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68) evicted. New cstate: current_term: 2 leader_uuid: "36de6d0f384b45f995635dfb983c6fa7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } health_report { overall_health: HEALTHY } } }
I20260505 20:19:01.562333 649 tablet_service.cc:1558] Processing DeleteTablet for tablet 71bd71769aa04036ab6a88793b08e0ca with delete_type TABLET_DATA_TOMBSTONED (TS b4b867020b7d41db8d83c3a52ef5a127 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50058
I20260505 20:19:01.566347 895 ts_tablet_manager.cc:1916] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c692829e03fc452ca8ff8ec11129cc61 | 127.27.255.126:39413 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
I20260505 20:19:01.567215 436 consensus_queue.cc:237] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: NON_VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: true } }
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | HEALTHY | <none> | 0 | 0
36de6d0f384b45f995635dfb983c6fa7 | 127.27.255.65:46287 | HEALTHY | <none> | 1 | 0
a5b2e66d4e6645daaa190ab40f2def19 | 127.27.255.67:36591 | HEALTHY | <none> | 0 | 0
b4b867020b7d41db8d83c3a52ef5a127 | 127.27.255.68:37635 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 71bd71769aa04036ab6a88793b08e0ca of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
a5b2e66d4e6645daaa190ab40f2def19 (127.27.255.67:36591): RUNNING
36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65:46287): RUNNING [LEADER]
b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = a5b2e66d4e6645daaa190ab40f2def19
B = 36de6d0f384b45f995635dfb983c6fa7
C = b4b867020b7d41db8d83c3a52ef5a127
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
I20260505 20:19:01.577348 585 raft_consensus.cc:1275] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 2 FOLLOWER]: Refusing update from remote peer 36de6d0f384b45f995635dfb983c6fa7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
master | A B* C | | | Yes
A | A B C | 2 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
I20260505 20:19:01.579456 850 consensus_queue.cc:1048] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260505 20:19:01.590730 895 ts_tablet_manager.cc:1929] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260505 20:19:01.591427 895 log.cc:1199] T 71bd71769aa04036ab6a88793b08e0ca P b4b867020b7d41db8d83c3a52ef5a127: Deleting WAL directory at /tmp/dist-test-taskQgcEo8/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1778012272869005-28669-0/minicluster-data/ts-3-root/wals/71bd71769aa04036ab6a88793b08e0ca
I20260505 20:19:01.591028 856 raft_consensus.cc:2955] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 2d71a29bf9bc444fa630af946677a4c3 (127.27.255.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: NON_VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: true } } }
I20260505 20:19:01.595070 318 catalog_manager.cc:5002] TS b4b867020b7d41db8d83c3a52ef5a127 (127.27.255.68:37635): tablet 71bd71769aa04036ab6a88793b08e0ca (table test-workload [id=280a291591584e8387725db070ed5a4d]) successfully deleted
W20260505 20:19:01.600466 402 consensus_peers.cc:597] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 -> Peer 2d71a29bf9bc444fa630af946677a4c3 (127.27.255.66:45409): Couldn't send request to peer 2d71a29bf9bc444fa630af946677a4c3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71bd71769aa04036ab6a88793b08e0ca. This is attempt 1: this message will repeat every 5th retry.
I20260505 20:19:01.597126 585 raft_consensus.cc:2955] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER 2d71a29bf9bc444fa630af946677a4c3 (127.27.255.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: NON_VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: true } } }
I20260505 20:19:01.602671 28669 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260505 20:19:01.608639 321 catalog_manager.cc:5184] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 71bd71769aa04036ab6a88793b08e0ca with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260505 20:19:01.613704 331 catalog_manager.cc:5671] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 reported cstate change: config changed from index 3 to 4, NON_VOTER 2d71a29bf9bc444fa630af946677a4c3 (127.27.255.66) added. New cstate: current_term: 2 leader_uuid: "36de6d0f384b45f995635dfb983c6fa7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: NON_VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260505 20:19:02.012745 904 ts_tablet_manager.cc:933] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: Initiating tablet copy from peer 36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65:46287)
I20260505 20:19:02.014734 904 tablet_copy_client.cc:323] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: tablet copy: Beginning tablet copy session from remote peer at address 127.27.255.65:46287
I20260505 20:19:02.029202 446 tablet_copy_service.cc:140] P 36de6d0f384b45f995635dfb983c6fa7: Received BeginTabletCopySession request for tablet 71bd71769aa04036ab6a88793b08e0ca from peer 2d71a29bf9bc444fa630af946677a4c3 ({username='slave'} at 127.0.0.1:37240)
I20260505 20:19:02.029948 446 tablet_copy_service.cc:161] P 36de6d0f384b45f995635dfb983c6fa7: Beginning new tablet copy session on tablet 71bd71769aa04036ab6a88793b08e0ca from peer 2d71a29bf9bc444fa630af946677a4c3 at {username='slave'} at 127.0.0.1:37240: session id = 2d71a29bf9bc444fa630af946677a4c3-71bd71769aa04036ab6a88793b08e0ca
I20260505 20:19:02.036716 446 tablet_copy_source_session.cc:215] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7: Tablet Copy: opened 0 blocks and 1 log segments
I20260505 20:19:02.041109 904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71bd71769aa04036ab6a88793b08e0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 20:19:02.056674 904 tablet_copy_client.cc:806] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: tablet copy: Starting download of 0 data blocks...
I20260505 20:19:02.057471 904 tablet_copy_client.cc:670] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: tablet copy: Starting download of 1 WAL segments...
I20260505 20:19:02.061978 904 tablet_copy_client.cc:538] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260505 20:19:02.070858 904 tablet_bootstrap.cc:492] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: Bootstrap starting.
I20260505 20:19:02.094918 904 tablet_bootstrap.cc:492] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 20:19:02.095988 904 tablet_bootstrap.cc:492] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: Bootstrap complete.
I20260505 20:19:02.097014 904 ts_tablet_manager.cc:1403] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.009s
I20260505 20:19:02.100319 904 raft_consensus.cc:359] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: NON_VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: true } }
I20260505 20:19:02.101051 904 raft_consensus.cc:740] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2d71a29bf9bc444fa630af946677a4c3, State: Initialized, Role: LEARNER
I20260505 20:19:02.101866 904 consensus_queue.cc:260] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: NON_VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: true } }
I20260505 20:19:02.104539 904 ts_tablet_manager.cc:1434] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260505 20:19:02.106696 446 tablet_copy_service.cc:342] P 36de6d0f384b45f995635dfb983c6fa7: Request end of tablet copy session 2d71a29bf9bc444fa630af946677a4c3-71bd71769aa04036ab6a88793b08e0ca received from {username='slave'} at 127.0.0.1:37240
I20260505 20:19:02.107112 446 tablet_copy_service.cc:434] P 36de6d0f384b45f995635dfb983c6fa7: ending tablet copy session 2d71a29bf9bc444fa630af946677a4c3-71bd71769aa04036ab6a88793b08e0ca on tablet 71bd71769aa04036ab6a88793b08e0ca with peer 2d71a29bf9bc444fa630af946677a4c3
I20260505 20:19:02.233660 426 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260505 20:19:02.237931 501 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:02.249389 649 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:02.250628 575 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c692829e03fc452ca8ff8ec11129cc61 | 127.27.255.126:39413 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | HEALTHY | <none> | 0 | 0
36de6d0f384b45f995635dfb983c6fa7 | 127.27.255.65:46287 | HEALTHY | <none> | 1 | 0
a5b2e66d4e6645daaa190ab40f2def19 | 127.27.255.67:36591 | HEALTHY | <none> | 0 | 0
b4b867020b7d41db8d83c3a52ef5a127 | 127.27.255.68:37635 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 71bd71769aa04036ab6a88793b08e0ca of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
a5b2e66d4e6645daaa190ab40f2def19 (127.27.255.67:36591): RUNNING
36de6d0f384b45f995635dfb983c6fa7 (127.27.255.65:46287): RUNNING [LEADER]
2d71a29bf9bc444fa630af946677a4c3 (127.27.255.66:45409): RUNNING [NONVOTER]
All reported replicas are:
A = a5b2e66d4e6645daaa190ab40f2def19
B = 36de6d0f384b45f995635dfb983c6fa7
C = 2d71a29bf9bc444fa630af946677a4c3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 2 | 4 | Yes
B | A B* C~ | 2 | 4 | Yes
C | A B C~ | 2 | 4 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | b4b867020b7d41db8d83c3a52ef5a127 | 127.27.255.68:37635 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260505 20:19:02.522507 28669 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260505 20:19:02.590101 511 raft_consensus.cc:1217] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[]
I20260505 20:19:03.198190 933 raft_consensus.cc:1064] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7: attempting to promote NON_VOTER 2d71a29bf9bc444fa630af946677a4c3 to VOTER
I20260505 20:19:03.200371 933 consensus_queue.cc:237] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: false } }
I20260505 20:19:03.207360 511 raft_consensus.cc:1275] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3 [term 2 LEARNER]: Refusing update from remote peer 36de6d0f384b45f995635dfb983c6fa7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260505 20:19:03.209384 946 consensus_queue.cc:1048] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260505 20:19:03.210340 585 raft_consensus.cc:1275] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 2 FOLLOWER]: Refusing update from remote peer 36de6d0f384b45f995635dfb983c6fa7: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260505 20:19:03.212112 946 consensus_queue.cc:1048] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260505 20:19:03.220225 933 raft_consensus.cc:2955] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, 2d71a29bf9bc444fa630af946677a4c3 (127.27.255.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: false } } }
I20260505 20:19:03.223276 511 raft_consensus.cc:2955] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, 2d71a29bf9bc444fa630af946677a4c3 (127.27.255.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: false } } }
I20260505 20:19:03.240589 332 catalog_manager.cc:5671] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 reported cstate change: config changed from index 4 to 5, 2d71a29bf9bc444fa630af946677a4c3 (127.27.255.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "36de6d0f384b45f995635dfb983c6fa7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260505 20:19:03.241621 585 raft_consensus.cc:2955] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, 2d71a29bf9bc444fa630af946677a4c3 (127.27.255.66) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5b2e66d4e6645daaa190ab40f2def19" member_type: VOTER last_known_addr { host: "127.27.255.67" port: 36591 } } peers { permanent_uuid: "36de6d0f384b45f995635dfb983c6fa7" member_type: VOTER last_known_addr { host: "127.27.255.65" port: 46287 } } peers { permanent_uuid: "2d71a29bf9bc444fa630af946677a4c3" member_type: VOTER last_known_addr { host: "127.27.255.66" port: 45409 } attrs { promote: false } } }
I20260505 20:19:03.332371 426 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260505 20:19:03.333774 501 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:03.338896 575 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260505 20:19:03.359819 649 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
c692829e03fc452ca8ff8ec11129cc61 | 127.27.255.126:39413 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2d71a29bf9bc444fa630af946677a4c3 | 127.27.255.66:45409 | HEALTHY | <none> | 0 | 0
36de6d0f384b45f995635dfb983c6fa7 | 127.27.255.65:46287 | HEALTHY | <none> | 1 | 0
a5b2e66d4e6645daaa190ab40f2def19 | 127.27.255.67:36591 | HEALTHY | <none> | 0 | 0
b4b867020b7d41db8d83c3a52ef5a127 | 127.27.255.68:37635 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | b4b867020b7d41db8d83c3a52ef5a127 | 127.27.255.68:37635 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260505 20:19:03.631361 28669 log_verifier.cc:126] Checking tablet 71bd71769aa04036ab6a88793b08e0ca
I20260505 20:19:03.709185 28669 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 71bd71769aa04036ab6a88793b08e0ca
I20260505 20:19:03.711659 28669 tablet_server.cc:179] TabletServer@127.27.255.65:0 shutting down...
I20260505 20:19:03.748956 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:19:03.750279 28669 tablet_replica.cc:333] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7: stopping tablet replica
I20260505 20:19:03.751173 28669 raft_consensus.cc:2243] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 2 LEADER]: Raft consensus shutting down.
I20260505 20:19:03.752139 28669 raft_consensus.cc:2272] T 71bd71769aa04036ab6a88793b08e0ca P 36de6d0f384b45f995635dfb983c6fa7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:19:03.776417 28669 tablet_server.cc:196] TabletServer@127.27.255.65:0 shutdown complete.
I20260505 20:19:03.788395 28669 tablet_server.cc:179] TabletServer@127.27.255.66:0 shutting down...
I20260505 20:19:03.813208 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:19:03.813994 28669 tablet_replica.cc:333] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3: stopping tablet replica
I20260505 20:19:03.814572 28669 raft_consensus.cc:2243] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260505 20:19:03.815125 28669 raft_consensus.cc:2272] T 71bd71769aa04036ab6a88793b08e0ca P 2d71a29bf9bc444fa630af946677a4c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:19:03.838495 28669 tablet_server.cc:196] TabletServer@127.27.255.66:0 shutdown complete.
I20260505 20:19:03.850737 28669 tablet_server.cc:179] TabletServer@127.27.255.67:0 shutting down...
I20260505 20:19:03.877203 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:19:03.877991 28669 tablet_replica.cc:333] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19: stopping tablet replica
I20260505 20:19:03.878582 28669 raft_consensus.cc:2243] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260505 20:19:03.879297 28669 raft_consensus.cc:2272] T 71bd71769aa04036ab6a88793b08e0ca P a5b2e66d4e6645daaa190ab40f2def19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260505 20:19:03.902354 28669 tablet_server.cc:196] TabletServer@127.27.255.67:0 shutdown complete.
I20260505 20:19:03.913864 28669 tablet_server.cc:179] TabletServer@127.27.255.68:0 shutting down...
I20260505 20:19:03.976297 28669 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260505 20:19:03.998471 28669 tablet_server.cc:196] TabletServer@127.27.255.68:0 shutdown complete.
I20260505 20:19:04.017781 28669 master.cc:562] Master@127.27.255.126:39413 shutting down...
I20260505 20:19:04.045320 28669 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 1 LEADER]: Raft consensus shutting down.
I20260505 20:19:04.046387 28669 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260505 20:19:04.046823 28669 tablet_replica.cc:333] T 00000000000000000000000000000000 P c692829e03fc452ca8ff8ec11129cc61: stopping tablet replica
I20260505 20:19:04.070681 28669 master.cc:584] Master@127.27.255.126:39413 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (7127 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (12253 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (71079 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20260505 20:19:04.131556 28669 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~13 seconds ago
I20260505 20:19:04.132040 28669 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 8 messages since previous log ~17 seconds ago
I20260505 20:19:04.132263 28669 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 97 messages since previous log ~62 seconds ago
I20260505 20:19:04.132604 28669 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2404: suppressed but not reported on 9 messages since previous log ~16 seconds ago