[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260321 08:56:48.849000 24578 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.082s user 0.454s sys 0.627s
W20260321 08:56:48.849640 24578 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.083s user 0.454s sys 0.627s
I20260321 08:56:48.852517 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:44057
I20260321 08:56:48.854785 24578 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260321 08:56:48.856472 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:48.871002 24589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:48.872658 24590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:48.964102 24578 server_base.cc:1061] running on GCE node
W20260321 08:56:48.964339 24592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:48.965629 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:48.965850 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:48.966020 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083408966003 us; error 0 us; skew 500 ppm
I20260321 08:56:48.966697 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:48.971246 24578 webserver.cc:492] Webserver started at http://127.24.0.190:38371/ using document root <none> and password file <none>
I20260321 08:56:48.972023 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:48.972213 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:48.972581 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:48.977788 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "6038795d97884551ac2ef5250d0368c9"
format_stamp: "Formatted at 2026-03-21 08:56:48 on dist-test-slave-p54g"
I20260321 08:56:48.984866 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260321 08:56:48.989424 24597 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:48.990545 24578 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260321 08:56:48.990868 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "6038795d97884551ac2ef5250d0368c9"
format_stamp: "Formatted at 2026-03-21 08:56:48 on dist-test-slave-p54g"
I20260321 08:56:48.991185 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:49.007695 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:49.009094 24578 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260321 08:56:49.009503 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:49.087317 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:44057
I20260321 08:56:49.087379 24648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:44057 every 8 connection(s)
I20260321 08:56:49.093094 24649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:49.113550 24649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9: Bootstrap starting.
I20260321 08:56:49.119372 24649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:49.121114 24649 log.cc:826] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9: Log is configured to *not* fsync() on all Append() calls
I20260321 08:56:49.125224 24649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9: No bootstrap required, opened a new log
I20260321 08:56:49.145006 24649 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038795d97884551ac2ef5250d0368c9" member_type: VOTER }
I20260321 08:56:49.145480 24649 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:56:49.145673 24649 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6038795d97884551ac2ef5250d0368c9, State: Initialized, Role: FOLLOWER
I20260321 08:56:49.146399 24649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038795d97884551ac2ef5250d0368c9" member_type: VOTER }
I20260321 08:56:49.146840 24649 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:56:49.147068 24649 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:56:49.147337 24649 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:49.151050 24649 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038795d97884551ac2ef5250d0368c9" member_type: VOTER }
I20260321 08:56:49.151636 24649 leader_election.cc:304] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6038795d97884551ac2ef5250d0368c9; no voters:
I20260321 08:56:49.153221 24649 leader_election.cc:290] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:56:49.153575 24652 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:56:49.155737 24652 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 1 LEADER]: Becoming Leader. State: Replica: 6038795d97884551ac2ef5250d0368c9, State: Running, Role: LEADER
I20260321 08:56:49.156500 24652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038795d97884551ac2ef5250d0368c9" member_type: VOTER }
I20260321 08:56:49.157496 24649 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:56:49.163825 24654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6038795d97884551ac2ef5250d0368c9. Latest consensus state: current_term: 1 leader_uuid: "6038795d97884551ac2ef5250d0368c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038795d97884551ac2ef5250d0368c9" member_type: VOTER } }
I20260321 08:56:49.164494 24654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [sys.catalog]: This master's current role is: LEADER
I20260321 08:56:49.166956 24653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6038795d97884551ac2ef5250d0368c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038795d97884551ac2ef5250d0368c9" member_type: VOTER } }
I20260321 08:56:49.167500 24653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [sys.catalog]: This master's current role is: LEADER
I20260321 08:56:49.172784 24661 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:56:49.182363 24661 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:56:49.188099 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:56:49.198834 24661 catalog_manager.cc:1357] Generated new cluster ID: 74bd0f183de747ef9a1e7aee63a747ba
I20260321 08:56:49.199070 24661 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:56:49.237564 24661 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:56:49.239027 24661 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:56:49.253773 24661 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9: Generated new TSK 0
I20260321 08:56:49.254776 24661 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:56:49.318579 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:49.326262 24671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:49.327582 24672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:49.331266 24674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:49.331454 24578 server_base.cc:1061] running on GCE node
I20260321 08:56:49.332386 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:49.332635 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:49.332829 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083409332813 us; error 0 us; skew 500 ppm
I20260321 08:56:49.333567 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:49.337420 24578 webserver.cc:492] Webserver started at http://127.24.0.129:41571/ using document root <none> and password file <none>
I20260321 08:56:49.338075 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:49.338330 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:49.338654 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:49.340247 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "524401f2c0964981b55fa87e3e515877"
format_stamp: "Formatted at 2026-03-21 08:56:49 on dist-test-slave-p54g"
I20260321 08:56:49.346753 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260321 08:56:49.351531 24679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:49.352387 24578 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260321 08:56:49.352703 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "524401f2c0964981b55fa87e3e515877"
format_stamp: "Formatted at 2026-03-21 08:56:49 on dist-test-slave-p54g"
I20260321 08:56:49.353063 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:49.382210 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:49.383867 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:49.386566 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:56:49.391954 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:56:49.392271 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260321 08:56:49.392586 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:56:49.392812 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:49.477324 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:45635
I20260321 08:56:49.477381 24741 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:45635 every 8 connection(s)
I20260321 08:56:49.484671 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:49.496349 24746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:49.502303 24747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:49.507462 24749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:49.508020 24578 server_base.cc:1061] running on GCE node
I20260321 08:56:49.508898 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:49.509096 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:49.509255 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083409509233 us; error 0 us; skew 500 ppm
I20260321 08:56:49.509745 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:49.512902 24578 webserver.cc:492] Webserver started at http://127.24.0.130:35185/ using document root <none> and password file <none>
I20260321 08:56:49.513464 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:49.513650 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:49.513899 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:49.515292 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/ts-1-root/instance:
uuid: "60678d6519094ee9b2323cdec3f52d34"
format_stamp: "Formatted at 2026-03-21 08:56:49 on dist-test-slave-p54g"
I20260321 08:56:49.515455 24742 heartbeater.cc:344] Connected to a master server at 127.24.0.190:44057
I20260321 08:56:49.516029 24742 heartbeater.cc:461] Registering TS with master...
I20260321 08:56:49.517134 24742 heartbeater.cc:507] Master 127.24.0.190:44057 requested a full tablet report, sending...
I20260321 08:56:49.520676 24614 ts_manager.cc:194] Registered new tserver with Master: 524401f2c0964981b55fa87e3e515877 (127.24.0.129:45635)
I20260321 08:56:49.521899 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260321 08:56:49.523574 24614 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37424
I20260321 08:56:49.527097 24754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:49.527933 24578 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260321 08:56:49.528264 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/ts-1-root
uuid: "60678d6519094ee9b2323cdec3f52d34"
format_stamp: "Formatted at 2026-03-21 08:56:49 on dist-test-slave-p54g"
I20260321 08:56:49.528610 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1774083407564530-24578-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:49.544873 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:49.546504 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:49.549007 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:56:49.552570 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:56:49.552832 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:49.553110 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:56:49.553323 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:49.627270 24742 heartbeater.cc:499] Master 127.24.0.190:44057 was elected leader, sending a full tablet report...
I20260321 08:56:49.640444 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.130:36811
I20260321 08:56:49.640604 24816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.130:36811 every 8 connection(s)
I20260321 08:56:49.655766 24817 heartbeater.cc:344] Connected to a master server at 127.24.0.190:44057
I20260321 08:56:49.656164 24817 heartbeater.cc:461] Registering TS with master...
I20260321 08:56:49.656925 24817 heartbeater.cc:507] Master 127.24.0.190:44057 requested a full tablet report, sending...
I20260321 08:56:49.658694 24614 ts_manager.cc:194] Registered new tserver with Master: 60678d6519094ee9b2323cdec3f52d34 (127.24.0.130:36811)
I20260321 08:56:49.659178 24578 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.014511241s
I20260321 08:56:49.660311 24614 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37426
I20260321 08:56:49.688319 24614 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20260321 08:56:49.705704 24614 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260321 08:56:49.753018 24781 tablet_service.cc:1511] Processing CreateTablet for tablet aa3efca1b774422096a9cce6d9a7e6a2 (DEFAULT_TABLE table=test-table [id=79dd2ab41ccc4ec29a1c5d56366367cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260321 08:56:49.753017 24782 tablet_service.cc:1511] Processing CreateTablet for tablet 1475872227874aba98bacbeb82ba2df3 (DEFAULT_TABLE table=test-table [id=79dd2ab41ccc4ec29a1c5d56366367cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260321 08:56:49.754801 24781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3efca1b774422096a9cce6d9a7e6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:49.755631 24782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1475872227874aba98bacbeb82ba2df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:49.762853 24817 heartbeater.cc:499] Master 127.24.0.190:44057 was elected leader, sending a full tablet report...
I20260321 08:56:49.765841 24707 tablet_service.cc:1511] Processing CreateTablet for tablet 1475872227874aba98bacbeb82ba2df3 (DEFAULT_TABLE table=test-table [id=79dd2ab41ccc4ec29a1c5d56366367cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260321 08:56:49.767015 24707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1475872227874aba98bacbeb82ba2df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:49.773356 24706 tablet_service.cc:1511] Processing CreateTablet for tablet aa3efca1b774422096a9cce6d9a7e6a2 (DEFAULT_TABLE table=test-table [id=79dd2ab41ccc4ec29a1c5d56366367cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260321 08:56:49.774592 24706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa3efca1b774422096a9cce6d9a7e6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:49.778116 24830 tablet_bootstrap.cc:492] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34: Bootstrap starting.
I20260321 08:56:49.785914 24830 tablet_bootstrap.cc:654] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:49.789350 24831 tablet_bootstrap.cc:492] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877: Bootstrap starting.
I20260321 08:56:49.793784 24830 tablet_bootstrap.cc:492] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34: No bootstrap required, opened a new log
I20260321 08:56:49.794139 24830 ts_tablet_manager.cc:1403] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260321 08:56:49.796041 24830 raft_consensus.cc:359] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:49.796495 24830 raft_consensus.cc:740] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60678d6519094ee9b2323cdec3f52d34, State: Initialized, Role: FOLLOWER
I20260321 08:56:49.796996 24831 tablet_bootstrap.cc:654] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:49.796940 24830 consensus_queue.cc:260] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:49.800266 24830 ts_tablet_manager.cc:1434] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260321 08:56:49.801193 24830 tablet_bootstrap.cc:492] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34: Bootstrap starting.
I20260321 08:56:49.806934 24830 tablet_bootstrap.cc:654] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:49.817821 24831 tablet_bootstrap.cc:492] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877: No bootstrap required, opened a new log
I20260321 08:56:49.818328 24831 ts_tablet_manager.cc:1403] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.013s
I20260321 08:56:49.818352 24830 tablet_bootstrap.cc:492] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34: No bootstrap required, opened a new log
I20260321 08:56:49.818943 24830 ts_tablet_manager.cc:1403] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.006s
I20260321 08:56:49.821543 24830 raft_consensus.cc:359] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } } peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } }
I20260321 08:56:49.822208 24830 raft_consensus.cc:740] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60678d6519094ee9b2323cdec3f52d34, State: Initialized, Role: FOLLOWER
I20260321 08:56:49.822582 24831 raft_consensus.cc:359] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } } peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } }
I20260321 08:56:49.822788 24830 consensus_queue.cc:260] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } } peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } }
I20260321 08:56:49.823240 24831 raft_consensus.cc:740] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524401f2c0964981b55fa87e3e515877, State: Initialized, Role: FOLLOWER
I20260321 08:56:49.823855 24831 consensus_queue.cc:260] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } } peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } }
I20260321 08:56:49.825258 24830 ts_tablet_manager.cc:1434] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260321 08:56:49.827594 24831 ts_tablet_manager.cc:1434] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260321 08:56:49.828421 24831 tablet_bootstrap.cc:492] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877: Bootstrap starting.
I20260321 08:56:49.834120 24831 tablet_bootstrap.cc:654] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:49.849077 24831 tablet_bootstrap.cc:492] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877: No bootstrap required, opened a new log
I20260321 08:56:49.849543 24831 ts_tablet_manager.cc:1403] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20260321 08:56:49.852185 24831 raft_consensus.cc:359] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:49.852816 24831 raft_consensus.cc:740] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524401f2c0964981b55fa87e3e515877, State: Initialized, Role: FOLLOWER
I20260321 08:56:49.853449 24831 consensus_queue.cc:260] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:49.855782 24831 ts_tablet_manager.cc:1434] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260321 08:56:49.866627 24578 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20260321 08:56:49.866910 24578 raft_consensus.cc:423] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 0 FOLLOWER]: Emulating election...
I20260321 08:56:49.867122 24578 raft_consensus.cc:3060] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:49.870754 24578 raft_consensus.cc:697] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 1 LEADER]: Becoming Leader. State: Replica: 60678d6519094ee9b2323cdec3f52d34, State: Running, Role: LEADER
I20260321 08:56:49.871309 24578 consensus_queue.cc:237] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } } peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } }
I20260321 08:56:49.874562 24578 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260321 08:56:49.880872 24612 catalog_manager.cc:5671] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60678d6519094ee9b2323cdec3f52d34 (127.24.0.130). New cstate: current_term: 1 leader_uuid: "60678d6519094ee9b2323cdec3f52d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } health_report { overall_health: UNKNOWN } } }
W20260321 08:56:49.895403 24818 tablet.cc:2379] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260321 08:56:49.897441 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260321 08:56:50.001029 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260321 08:56:50.204438 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260321 08:56:50.471989 24717 raft_consensus.cc:3060] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:50.481428 24841 consensus_queue.cc:1048] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:56:50.507946 24578 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260321 08:56:50.508183 24578 raft_consensus.cc:423] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 0 FOLLOWER]: Emulating election...
I20260321 08:56:50.508394 24578 raft_consensus.cc:3060] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:50.511866 24578 raft_consensus.cc:697] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 1 LEADER]: Becoming Leader. State: Replica: 60678d6519094ee9b2323cdec3f52d34, State: Running, Role: LEADER
I20260321 08:56:50.512480 24578 consensus_queue.cc:237] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:50.514786 24578 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260321 08:56:50.518333 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260321 08:56:50.518682 24613 catalog_manager.cc:5671] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60678d6519094ee9b2323cdec3f52d34 (127.24.0.130). New cstate: current_term: 1 leader_uuid: "60678d6519094ee9b2323cdec3f52d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } health_report { overall_health: HEALTHY } } }
I20260321 08:56:50.621778 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260321 08:56:50.825670 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260321 08:56:51.123531 24717 raft_consensus.cc:3060] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:51.129707 24853 consensus_queue.cc:1048] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:56:51.133617 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260321 08:56:51.737093 24578 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260321 08:56:51.737347 24578 raft_consensus.cc:423] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 1 LEADER]: Emulating election...
I20260321 08:56:51.737562 24578 raft_consensus.cc:3055] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 08:56:51.737749 24578 raft_consensus.cc:740] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60678d6519094ee9b2323cdec3f52d34, State: Running, Role: LEADER
I20260321 08:56:51.738219 24578 consensus_queue.cc:260] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:51.738965 24578 raft_consensus.cc:3060] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:56:51.742861 24578 raft_consensus.cc:697] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 2 LEADER]: Becoming Leader. State: Replica: 60678d6519094ee9b2323cdec3f52d34, State: Running, Role: LEADER
I20260321 08:56:51.743394 24578 consensus_queue.cc:237] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:51.744870 24578 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260321 08:56:51.748095 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260321 08:56:51.747814 24613 catalog_manager.cc:5671] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "60678d6519094ee9b2323cdec3f52d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } health_report { overall_health: HEALTHY } } }
I20260321 08:56:51.851562 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260321 08:56:52.055289 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260321 08:56:52.356822 24716 raft_consensus.cc:3060] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:56:52.358809 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260321 08:56:52.361744 24716 raft_consensus.cc:1275] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 2 FOLLOWER]: Refusing update from remote peer 60678d6519094ee9b2323cdec3f52d34: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 08:56:52.362794 24856 consensus_queue.cc:1048] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 08:56:52.761988 24578 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260321 08:56:52.762223 24578 raft_consensus.cc:423] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 2 FOLLOWER]: Emulating election...
I20260321 08:56:52.762428 24578 raft_consensus.cc:3060] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:56:52.766146 24578 raft_consensus.cc:697] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 3 LEADER]: Becoming Leader. State: Replica: 524401f2c0964981b55fa87e3e515877, State: Running, Role: LEADER
I20260321 08:56:52.766772 24578 consensus_queue.cc:237] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:52.768548 24578 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260321 08:56:52.771795 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260321 08:56:52.771795 24613 catalog_manager.cc:5671] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 reported cstate change: term changed from 2 to 3, leader changed from 60678d6519094ee9b2323cdec3f52d34 (127.24.0.130) to 524401f2c0964981b55fa87e3e515877 (127.24.0.129). New cstate: current_term: 3 leader_uuid: "524401f2c0964981b55fa87e3e515877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } health_report { overall_health: UNKNOWN } } }
I20260321 08:56:52.875257 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260321 08:56:53.078877 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260321 08:56:53.158936 24716 raft_consensus.cc:1240] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 3 LEADER]: Rejecting Update request from peer 60678d6519094ee9b2323cdec3f52d34 for earlier term 2. Current term is 3. Ops: []
I20260321 08:56:53.160053 24865 consensus_queue.cc:1059] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 08:56:53.160993 24865 consensus_queue.cc:1243] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Peer 524401f2c0964981b55fa87e3e515877 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260321 08:56:53.161314 24872 raft_consensus.cc:3055] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 08:56:53.161569 24872 raft_consensus.cc:740] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60678d6519094ee9b2323cdec3f52d34, State: Running, Role: LEADER
I20260321 08:56:53.162025 24872 consensus_queue.cc:260] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:53.162655 24872 raft_consensus.cc:3060] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:56:53.381479 24792 raft_consensus.cc:1275] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 3 FOLLOWER]: Refusing update from remote peer 524401f2c0964981b55fa87e3e515877: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 08:56:53.382735 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260321 08:56:53.382791 24873 consensus_queue.cc:1048] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 08:56:53.986261 24578 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20260321 08:56:53.986514 24578 raft_consensus.cc:423] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [term 1 FOLLOWER]: Emulating election...
I20260321 08:56:53.986739 24578 raft_consensus.cc:3060] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:56:53.990425 24578 raft_consensus.cc:697] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [term 2 LEADER]: Becoming Leader. State: Replica: 524401f2c0964981b55fa87e3e515877, State: Running, Role: LEADER
I20260321 08:56:53.991042 24578 consensus_queue.cc:237] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } } peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } }
I20260321 08:56:53.992720 24578 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260321 08:56:53.996073 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260321 08:56:53.995752 24612 catalog_manager.cc:5671] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 reported cstate change: term changed from 1 to 2, leader changed from 60678d6519094ee9b2323cdec3f52d34 (127.24.0.130) to 524401f2c0964981b55fa87e3e515877 (127.24.0.129). New cstate: current_term: 2 leader_uuid: "524401f2c0964981b55fa87e3e515877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } health_report { overall_health: HEALTHY } } }
I20260321 08:56:54.099560 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260321 08:56:54.165169 24716 raft_consensus.cc:1240] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [term 2 LEADER]: Rejecting Update request from peer 60678d6519094ee9b2323cdec3f52d34 for earlier term 1. Current term is 2. Ops: []
I20260321 08:56:54.166217 24872 consensus_queue.cc:1059] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 08:56:54.167300 24872 consensus_queue.cc:1243] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Peer 524401f2c0964981b55fa87e3e515877 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260321 08:56:54.167631 24886 raft_consensus.cc:3055] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 08:56:54.167919 24886 raft_consensus.cc:740] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60678d6519094ee9b2323cdec3f52d34, State: Running, Role: LEADER
I20260321 08:56:54.168365 24886 consensus_queue.cc:260] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } } peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } }
I20260321 08:56:54.169028 24886 raft_consensus.cc:3060] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:56:54.302990 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260321 08:56:54.461931 24791 raft_consensus.cc:1275] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 2 FOLLOWER]: Refusing update from remote peer 524401f2c0964981b55fa87e3e515877: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 08:56:54.463163 24880 consensus_queue.cc:1048] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 08:56:54.606738 24578 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260321 08:56:54.607007 24578 raft_consensus.cc:423] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 3 FOLLOWER]: Emulating election...
I20260321 08:56:54.607283 24578 raft_consensus.cc:3060] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:56:54.610980 24578 raft_consensus.cc:697] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 4 LEADER]: Becoming Leader. State: Replica: 60678d6519094ee9b2323cdec3f52d34, State: Running, Role: LEADER
I20260321 08:56:54.611476 24578 consensus_queue.cc:237] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:54.613118 24578 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260321 08:56:54.616196 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260321 08:56:54.616453 24612 catalog_manager.cc:5671] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 reported cstate change: term changed from 3 to 4, leader changed from 524401f2c0964981b55fa87e3e515877 (127.24.0.129) to 60678d6519094ee9b2323cdec3f52d34 (127.24.0.130). New cstate: current_term: 4 leader_uuid: "60678d6519094ee9b2323cdec3f52d34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } health_report { overall_health: HEALTHY } } }
I20260321 08:56:54.719733 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260321 08:56:54.923403 24578 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260321 08:56:54.956607 24791 raft_consensus.cc:1240] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 4 LEADER]: Rejecting Update request from peer 524401f2c0964981b55fa87e3e515877 for earlier term 3. Current term is 4. Ops: []
I20260321 08:56:54.957683 24888 consensus_queue.cc:1059] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 08:56:54.958209 24888 consensus_queue.cc:1243] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [LEADER]: Peer 60678d6519094ee9b2323cdec3f52d34 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260321 08:56:54.958504 24880 raft_consensus.cc:3055] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 08:56:54.958766 24880 raft_consensus.cc:740] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 524401f2c0964981b55fa87e3e515877, State: Running, Role: LEADER
I20260321 08:56:54.959244 24880 consensus_queue.cc:260] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 } } peers { permanent_uuid: "60678d6519094ee9b2323cdec3f52d34" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 36811 } }
I20260321 08:56:54.959864 24880 raft_consensus.cc:3060] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:56:55.105449 24716 raft_consensus.cc:1275] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 4 FOLLOWER]: Refusing update from remote peer 60678d6519094ee9b2323cdec3f52d34: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 08:56:55.106570 24886 consensus_queue.cc:1048] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "524401f2c0964981b55fa87e3e515877" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 45635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 08:56:55.429574 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:56:55.446336 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:56:55.446898 24578 tablet_replica.cc:333] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877: stopping tablet replica
I20260321 08:56:55.447412 24578 raft_consensus.cc:2243] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 08:56:55.447789 24578 raft_consensus.cc:2272] T aa3efca1b774422096a9cce6d9a7e6a2 P 524401f2c0964981b55fa87e3e515877 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 08:56:55.449944 24578 tablet_replica.cc:333] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877: stopping tablet replica
I20260321 08:56:55.450333 24578 raft_consensus.cc:2243] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [term 2 LEADER]: Raft consensus shutting down.
I20260321 08:56:55.450809 24578 raft_consensus.cc:2272] T 1475872227874aba98bacbeb82ba2df3 P 524401f2c0964981b55fa87e3e515877 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 08:56:55.529846 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:56:55.539196 24578 tablet_server.cc:179] TabletServer@127.24.0.130:0 shutting down...
I20260321 08:56:55.554886 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:56:55.555454 24578 tablet_replica.cc:333] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34: stopping tablet replica
I20260321 08:56:55.555948 24578 raft_consensus.cc:2243] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 08:56:55.556327 24578 raft_consensus.cc:2272] T 1475872227874aba98bacbeb82ba2df3 P 60678d6519094ee9b2323cdec3f52d34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 08:56:55.557667 24578 tablet_replica.cc:333] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34: stopping tablet replica
I20260321 08:56:55.558071 24578 raft_consensus.cc:2243] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 4 LEADER]: Raft consensus shutting down.
I20260321 08:56:55.558560 24578 raft_consensus.cc:2272] T aa3efca1b774422096a9cce6d9a7e6a2 P 60678d6519094ee9b2323cdec3f52d34 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 08:56:55.576448 24578 tablet_server.cc:196] TabletServer@127.24.0.130:0 shutdown complete.
I20260321 08:56:55.584347 24578 master.cc:562] Master@127.24.0.190:44057 shutting down...
I20260321 08:56:55.596185 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:56:55.596633 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:56:55.596925 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 6038795d97884551ac2ef5250d0368c9: stopping tablet replica
I20260321 08:56:55.614547 24578 master.cc:584] Master@127.24.0.190:44057 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (7911 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20260321 08:56:55.638559 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:34309
I20260321 08:56:55.639478 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:55.643718 24903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:55.644695 24904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:55.647047 24906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:55.647186 24578 server_base.cc:1061] running on GCE node
I20260321 08:56:55.647987 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:55.648176 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:55.648319 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083415648302 us; error 0 us; skew 500 ppm
I20260321 08:56:55.648780 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:55.650807 24578 webserver.cc:492] Webserver started at http://127.24.0.190:33653/ using document root <none> and password file <none>
I20260321 08:56:55.651221 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:55.651382 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:55.651616 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:55.652583 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "ab8741ce6ae442a1ba831377be61b6e7"
format_stamp: "Formatted at 2026-03-21 08:56:55 on dist-test-slave-p54g"
I20260321 08:56:55.656427 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260321 08:56:55.659252 24911 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:55.659973 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260321 08:56:55.660200 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "ab8741ce6ae442a1ba831377be61b6e7"
format_stamp: "Formatted at 2026-03-21 08:56:55 on dist-test-slave-p54g"
I20260321 08:56:55.660418 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:55.671392 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:55.672271 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:55.704344 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:34309
I20260321 08:56:55.704422 24962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:34309 every 8 connection(s)
I20260321 08:56:55.707810 24963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:55.716777 24963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7: Bootstrap starting.
I20260321 08:56:55.720388 24963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:55.723742 24963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7: No bootstrap required, opened a new log
I20260321 08:56:55.725448 24963 raft_consensus.cc:359] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8741ce6ae442a1ba831377be61b6e7" member_type: VOTER }
I20260321 08:56:55.725797 24963 raft_consensus.cc:385] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:56:55.725981 24963 raft_consensus.cc:740] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8741ce6ae442a1ba831377be61b6e7, State: Initialized, Role: FOLLOWER
I20260321 08:56:55.726445 24963 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8741ce6ae442a1ba831377be61b6e7" member_type: VOTER }
I20260321 08:56:55.726855 24963 raft_consensus.cc:399] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:56:55.727054 24963 raft_consensus.cc:493] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:56:55.727265 24963 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:55.730937 24963 raft_consensus.cc:515] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8741ce6ae442a1ba831377be61b6e7" member_type: VOTER }
I20260321 08:56:55.731359 24963 leader_election.cc:304] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab8741ce6ae442a1ba831377be61b6e7; no voters:
I20260321 08:56:55.732352 24963 leader_election.cc:290] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:56:55.732609 24966 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:56:55.733748 24966 raft_consensus.cc:697] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 1 LEADER]: Becoming Leader. State: Replica: ab8741ce6ae442a1ba831377be61b6e7, State: Running, Role: LEADER
I20260321 08:56:55.734333 24966 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8741ce6ae442a1ba831377be61b6e7" member_type: VOTER }
I20260321 08:56:55.734963 24963 sys_catalog.cc:565] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:56:55.736763 24968 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab8741ce6ae442a1ba831377be61b6e7. Latest consensus state: current_term: 1 leader_uuid: "ab8741ce6ae442a1ba831377be61b6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8741ce6ae442a1ba831377be61b6e7" member_type: VOTER } }
I20260321 08:56:55.737258 24968 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [sys.catalog]: This master's current role is: LEADER
I20260321 08:56:55.737282 24967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab8741ce6ae442a1ba831377be61b6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8741ce6ae442a1ba831377be61b6e7" member_type: VOTER } }
I20260321 08:56:55.737840 24967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [sys.catalog]: This master's current role is: LEADER
I20260321 08:56:55.739778 24971 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:56:55.744482 24971 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:56:55.748382 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:56:55.752163 24971 catalog_manager.cc:1357] Generated new cluster ID: db902cd493b143bca939e3a32d461c34
I20260321 08:56:55.752367 24971 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:56:55.778110 24971 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:56:55.779186 24971 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:56:55.794358 24971 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7: Generated new TSK 0
I20260321 08:56:55.794833 24971 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:56:55.815214 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:55.820262 24984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:55.821322 24985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:55.822562 24987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:55.822975 24578 server_base.cc:1061] running on GCE node
I20260321 08:56:55.823675 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:55.823904 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:55.824059 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083415824042 us; error 0 us; skew 500 ppm
I20260321 08:56:55.824523 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:55.826525 24578 webserver.cc:492] Webserver started at http://127.24.0.129:45769/ using document root <none> and password file <none>
I20260321 08:56:55.826964 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:55.827137 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:55.827371 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:55.828382 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "d331cb62527245c3aa4bdaf9e78d061f"
format_stamp: "Formatted at 2026-03-21 08:56:55 on dist-test-slave-p54g"
I20260321 08:56:55.832427 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260321 08:56:55.835274 24992 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:55.835992 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:56:55.836234 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "d331cb62527245c3aa4bdaf9e78d061f"
format_stamp: "Formatted at 2026-03-21 08:56:55 on dist-test-slave-p54g"
I20260321 08:56:55.836503 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:55.845283 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:55.846190 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:55.847466 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:56:55.849577 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:56:55.849763 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:55.849980 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:56:55.850121 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:55.885854 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:38575
I20260321 08:56:55.885929 25054 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:38575 every 8 connection(s)
I20260321 08:56:55.889833 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:55.896224 25059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:55.897243 25060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:55.899978 24578 server_base.cc:1061] running on GCE node
W20260321 08:56:55.900120 25062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:55.900925 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:55.901127 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:55.901254 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083415901243 us; error 0 us; skew 500 ppm
I20260321 08:56:55.901687 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:55.904260 24578 webserver.cc:492] Webserver started at http://127.24.0.130:38299/ using document root <none> and password file <none>
I20260321 08:56:55.904778 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:55.904968 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:55.905278 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:55.905241 25055 heartbeater.cc:344] Connected to a master server at 127.24.0.190:34309
I20260321 08:56:55.905652 25055 heartbeater.cc:461] Registering TS with master...
I20260321 08:56:55.906270 25055 heartbeater.cc:507] Master 127.24.0.190:34309 requested a full tablet report, sending...
I20260321 08:56:55.906394 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/ts-1-root/instance:
uuid: "e1d5fe86a51f485ca8b11a0424ca6a89"
format_stamp: "Formatted at 2026-03-21 08:56:55 on dist-test-slave-p54g"
I20260321 08:56:55.908077 24928 ts_manager.cc:194] Registered new tserver with Master: d331cb62527245c3aa4bdaf9e78d061f (127.24.0.129:38575)
I20260321 08:56:55.909611 24928 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33828
I20260321 08:56:55.910892 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260321 08:56:55.913977 25067 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:55.914683 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260321 08:56:55.914932 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/ts-1-root
uuid: "e1d5fe86a51f485ca8b11a0424ca6a89"
format_stamp: "Formatted at 2026-03-21 08:56:55 on dist-test-slave-p54g"
I20260321 08:56:55.915205 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1774083407564530-24578-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:55.931404 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:55.932446 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:55.933740 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:56:55.935751 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:56:55.935966 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:55.936201 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:56:55.936362 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:55.983980 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.130:36769
I20260321 08:56:55.984058 25129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.130:36769 every 8 connection(s)
I20260321 08:56:55.999105 25130 heartbeater.cc:344] Connected to a master server at 127.24.0.190:34309
I20260321 08:56:55.999434 25130 heartbeater.cc:461] Registering TS with master...
I20260321 08:56:56.000128 25130 heartbeater.cc:507] Master 127.24.0.190:34309 requested a full tablet report, sending...
I20260321 08:56:56.001814 24928 ts_manager.cc:194] Registered new tserver with Master: e1d5fe86a51f485ca8b11a0424ca6a89 (127.24.0.130:36769)
I20260321 08:56:56.002916 24578 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.014686836s
I20260321 08:56:56.003023 24928 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33834
I20260321 08:56:56.011854 25055 heartbeater.cc:499] Master 127.24.0.190:34309 was elected leader, sending a full tablet report...
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:422: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20260321 08:56:56.023538 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:56:56.038439 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:56:56.053115 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:56:56.059250 24578 tablet_server.cc:179] TabletServer@127.24.0.130:0 shutting down...
I20260321 08:56:56.072193 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:56:56.086874 24578 tablet_server.cc:196] TabletServer@127.24.0.130:0 shutdown complete.
I20260321 08:56:56.093164 24578 master.cc:562] Master@127.24.0.190:34309 shutting down...
I20260321 08:56:56.104125 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:56:56.104552 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:56:56.104884 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P ab8741ce6ae442a1ba831377be61b6e7: stopping tablet replica
I20260321 08:56:56.121987 24578 master.cc:584] Master@127.24.0.190:34309 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (503 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20260321 08:56:56.142725 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:41417
I20260321 08:56:56.143697 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:56.147855 25143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:56.148787 25144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:56.150113 25146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:56.150490 24578 server_base.cc:1061] running on GCE node
I20260321 08:56:56.151058 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:56.151218 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:56.151325 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083416151314 us; error 0 us; skew 500 ppm
I20260321 08:56:56.151727 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:56.154062 24578 webserver.cc:492] Webserver started at http://127.24.0.190:33435/ using document root <none> and password file <none>
I20260321 08:56:56.154474 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:56.154632 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:56.154843 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:56.155776 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "45d67c348b164ec6b30da338ddb01fa9"
format_stamp: "Formatted at 2026-03-21 08:56:56 on dist-test-slave-p54g"
I20260321 08:56:56.159677 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260321 08:56:56.162667 25151 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:56.163326 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260321 08:56:56.163553 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "45d67c348b164ec6b30da338ddb01fa9"
format_stamp: "Formatted at 2026-03-21 08:56:56 on dist-test-slave-p54g"
I20260321 08:56:56.163782 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:56.189040 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:56.189936 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:56.221866 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:41417
I20260321 08:56:56.221942 25202 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:41417 every 8 connection(s)
I20260321 08:56:56.225282 25203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:56.234089 25203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9: Bootstrap starting.
I20260321 08:56:56.237814 25203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:56.241258 25203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9: No bootstrap required, opened a new log
I20260321 08:56:56.242993 25203 raft_consensus.cc:359] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d67c348b164ec6b30da338ddb01fa9" member_type: VOTER }
I20260321 08:56:56.243356 25203 raft_consensus.cc:385] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:56:56.243538 25203 raft_consensus.cc:740] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d67c348b164ec6b30da338ddb01fa9, State: Initialized, Role: FOLLOWER
I20260321 08:56:56.244016 25203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d67c348b164ec6b30da338ddb01fa9" member_type: VOTER }
I20260321 08:56:56.244410 25203 raft_consensus.cc:399] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:56:56.244598 25203 raft_consensus.cc:493] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:56:56.244797 25203 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:56.248386 25203 raft_consensus.cc:515] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d67c348b164ec6b30da338ddb01fa9" member_type: VOTER }
I20260321 08:56:56.248801 25203 leader_election.cc:304] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45d67c348b164ec6b30da338ddb01fa9; no voters:
I20260321 08:56:56.249752 25203 leader_election.cc:290] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:56:56.250021 25206 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:56:56.251125 25206 raft_consensus.cc:697] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 1 LEADER]: Becoming Leader. State: Replica: 45d67c348b164ec6b30da338ddb01fa9, State: Running, Role: LEADER
I20260321 08:56:56.251705 25206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d67c348b164ec6b30da338ddb01fa9" member_type: VOTER }
I20260321 08:56:56.252290 25203 sys_catalog.cc:565] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:56:56.254261 25207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45d67c348b164ec6b30da338ddb01fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d67c348b164ec6b30da338ddb01fa9" member_type: VOTER } }
I20260321 08:56:56.254315 25208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45d67c348b164ec6b30da338ddb01fa9. Latest consensus state: current_term: 1 leader_uuid: "45d67c348b164ec6b30da338ddb01fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45d67c348b164ec6b30da338ddb01fa9" member_type: VOTER } }
I20260321 08:56:56.254928 25208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [sys.catalog]: This master's current role is: LEADER
I20260321 08:56:56.254952 25207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [sys.catalog]: This master's current role is: LEADER
I20260321 08:56:56.257519 25212 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:56:56.261626 25212 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:56:56.265831 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:56:56.269258 25212 catalog_manager.cc:1357] Generated new cluster ID: 9eb832d1f46f4ee1b40b26e593cf82d7
I20260321 08:56:56.269524 25212 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:56:56.282213 25212 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:56:56.283320 25212 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:56:56.294315 25212 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9: Generated new TSK 0
I20260321 08:56:56.294852 25212 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:56:56.299913 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:56.304939 25224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:56.306248 25225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:56.306880 25227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:56.307281 24578 server_base.cc:1061] running on GCE node
I20260321 08:56:56.308002 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:56.308172 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:56.308288 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083416308278 us; error 0 us; skew 500 ppm
I20260321 08:56:56.308696 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:56.310940 24578 webserver.cc:492] Webserver started at http://127.24.0.129:44067/ using document root <none> and password file <none>
I20260321 08:56:56.311336 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:56.311482 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:56.311678 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:56.312644 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "bfcb88d0491b43828d68afbfd9b84979"
format_stamp: "Formatted at 2026-03-21 08:56:56 on dist-test-slave-p54g"
I20260321 08:56:56.316473 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260321 08:56:56.319299 25232 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:56.319972 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:56:56.320191 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "bfcb88d0491b43828d68afbfd9b84979"
format_stamp: "Formatted at 2026-03-21 08:56:56 on dist-test-slave-p54g"
I20260321 08:56:56.320415 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:56.359537 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:56.360532 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:56.361879 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:56:56.363957 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:56:56.364140 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:56.364362 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:56:56.364496 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:56.398906 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:38427
I20260321 08:56:56.398960 25294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:38427 every 8 connection(s)
I20260321 08:56:56.403508 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:56.409097 25299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:56.409695 25300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:56.416082 24578 server_base.cc:1061] running on GCE node
W20260321 08:56:56.416555 25303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:56.417420 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:56.417604 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:56.417757 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083416417740 us; error 0 us; skew 500 ppm
I20260321 08:56:56.418243 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:56.421208 24578 webserver.cc:492] Webserver started at http://127.24.0.130:43623/ using document root <none> and password file <none>
I20260321 08:56:56.421707 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:56.421666 25295 heartbeater.cc:344] Connected to a master server at 127.24.0.190:41417
I20260321 08:56:56.422021 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:56.422116 25295 heartbeater.cc:461] Registering TS with master...
I20260321 08:56:56.422292 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:56.422771 25295 heartbeater.cc:507] Master 127.24.0.190:41417 requested a full tablet report, sending...
I20260321 08:56:56.423372 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/ts-1-root/instance:
uuid: "1a51d601c2c949eea9e1ae1e67a33881"
format_stamp: "Formatted at 2026-03-21 08:56:56 on dist-test-slave-p54g"
I20260321 08:56:56.424626 25168 ts_manager.cc:194] Registered new tserver with Master: bfcb88d0491b43828d68afbfd9b84979 (127.24.0.129:38427)
I20260321 08:56:56.426153 25168 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54936
I20260321 08:56:56.427668 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260321 08:56:56.430915 25307 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:56.431663 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:56:56.431973 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/ts-1-root
uuid: "1a51d601c2c949eea9e1ae1e67a33881"
format_stamp: "Formatted at 2026-03-21 08:56:56 on dist-test-slave-p54g"
I20260321 08:56:56.432227 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1774083407564530-24578-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:56.462123 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:56.463038 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:56.464354 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:56:56.466336 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:56:56.466506 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:56.466725 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:56:56.466871 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:56.499843 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.130:34019
I20260321 08:56:56.499964 25369 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.130:34019 every 8 connection(s)
I20260321 08:56:56.511339 25370 heartbeater.cc:344] Connected to a master server at 127.24.0.190:41417
I20260321 08:56:56.511641 25370 heartbeater.cc:461] Registering TS with master...
I20260321 08:56:56.512295 25370 heartbeater.cc:507] Master 127.24.0.190:41417 requested a full tablet report, sending...
I20260321 08:56:56.513958 25168 ts_manager.cc:194] Registered new tserver with Master: 1a51d601c2c949eea9e1ae1e67a33881 (127.24.0.130:34019)
I20260321 08:56:56.514439 24578 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.011757862s
I20260321 08:56:56.515169 25168 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54944
I20260321 08:56:56.528605 25295 heartbeater.cc:499] Master 127.24.0.190:41417 was elected leader, sending a full tablet report...
I20260321 08:56:56.533522 25167 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260321 08:56:56.534888 25167 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260321 08:56:56.563342 25260 tablet_service.cc:1511] Processing CreateTablet for tablet b0e7be03a7a7457ca2c28feabfb00974 (DEFAULT_TABLE table=test-table [id=5f73305d19c54b0eab3b9f041b498646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260321 08:56:56.564395 25260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0e7be03a7a7457ca2c28feabfb00974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:56.564610 25335 tablet_service.cc:1511] Processing CreateTablet for tablet b0e7be03a7a7457ca2c28feabfb00974 (DEFAULT_TABLE table=test-table [id=5f73305d19c54b0eab3b9f041b498646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260321 08:56:56.565711 25335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0e7be03a7a7457ca2c28feabfb00974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:56.579221 25383 tablet_bootstrap.cc:492] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979: Bootstrap starting.
I20260321 08:56:56.580242 25384 tablet_bootstrap.cc:492] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881: Bootstrap starting.
I20260321 08:56:56.583477 25383 tablet_bootstrap.cc:654] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:56.584192 25384 tablet_bootstrap.cc:654] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:56.587602 25383 tablet_bootstrap.cc:492] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979: No bootstrap required, opened a new log
I20260321 08:56:56.588025 25383 ts_tablet_manager.cc:1403] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260321 08:56:56.588214 25384 tablet_bootstrap.cc:492] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881: No bootstrap required, opened a new log
I20260321 08:56:56.588614 25384 ts_tablet_manager.cc:1403] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260321 08:56:56.590318 25383 raft_consensus.cc:359] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcb88d0491b43828d68afbfd9b84979" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38427 } } peers { permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 } }
I20260321 08:56:56.590945 25383 raft_consensus.cc:740] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfcb88d0491b43828d68afbfd9b84979, State: Initialized, Role: FOLLOWER
I20260321 08:56:56.590785 25384 raft_consensus.cc:359] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcb88d0491b43828d68afbfd9b84979" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38427 } } peers { permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 } }
I20260321 08:56:56.591377 25384 raft_consensus.cc:740] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a51d601c2c949eea9e1ae1e67a33881, State: Initialized, Role: FOLLOWER
I20260321 08:56:56.591508 25383 consensus_queue.cc:260] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcb88d0491b43828d68afbfd9b84979" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38427 } } peers { permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 } }
I20260321 08:56:56.591953 25384 consensus_queue.cc:260] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcb88d0491b43828d68afbfd9b84979" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38427 } } peers { permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 } }
I20260321 08:56:56.594192 25383 ts_tablet_manager.cc:1434] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260321 08:56:56.594717 25370 heartbeater.cc:499] Master 127.24.0.190:41417 was elected leader, sending a full tablet report...
I20260321 08:56:56.595326 25384 ts_tablet_manager.cc:1434] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260321 08:56:56.622953 24578 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260321 08:56:56.623170 24578 raft_consensus.cc:423] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 0 FOLLOWER]: Emulating election...
I20260321 08:56:56.623386 24578 raft_consensus.cc:3060] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:56.627031 24578 raft_consensus.cc:697] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 1 LEADER]: Becoming Leader. State: Replica: bfcb88d0491b43828d68afbfd9b84979, State: Running, Role: LEADER
W20260321 08:56:56.627290 25295 heartbeater.cc:646] Failed to heartbeat to 127.24.0.190:41417 (0 consecutive failures): IO error: failing all heartbeats for tests
I20260321 08:56:56.627540 24578 consensus_queue.cc:237] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcb88d0491b43828d68afbfd9b84979" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38427 } } peers { permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 } }
I20260321 08:56:56.630194 24578 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260321 08:56:56.646221 24578 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260321 08:56:56.750134 24578 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260321 08:56:56.953713 24578 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260321 08:56:57.080651 25345 raft_consensus.cc:3060] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:57.086086 25387 consensus_queue.cc:1048] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:56:57.258713 24578 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260321 08:56:57.258935 24578 raft_consensus.cc:423] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 1 LEADER]: Emulating election...
I20260321 08:56:57.259104 24578 raft_consensus.cc:3055] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 08:56:57.259249 24578 raft_consensus.cc:740] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bfcb88d0491b43828d68afbfd9b84979, State: Running, Role: LEADER
I20260321 08:56:57.259603 24578 consensus_queue.cc:260] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcb88d0491b43828d68afbfd9b84979" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38427 } } peers { permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 } }
I20260321 08:56:57.260188 24578 raft_consensus.cc:3060] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:56:57.263746 24578 raft_consensus.cc:697] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 2 LEADER]: Becoming Leader. State: Replica: bfcb88d0491b43828d68afbfd9b84979, State: Running, Role: LEADER
I20260321 08:56:57.264283 24578 consensus_queue.cc:237] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcb88d0491b43828d68afbfd9b84979" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38427 } } peers { permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 } }
I20260321 08:56:57.265527 24578 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260321 08:56:57.268846 24578 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260321 08:56:57.372504 24578 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260321 08:56:57.576229 24578 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260321 08:56:57.813789 25345 raft_consensus.cc:3060] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:56:57.819516 25345 raft_consensus.cc:1275] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881 [term 2 FOLLOWER]: Refusing update from remote peer bfcb88d0491b43828d68afbfd9b84979: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 08:56:57.820776 25397 consensus_queue.cc:1048] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 08:56:57.881817 24578 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260321 08:56:57.882047 24578 raft_consensus.cc:423] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 2 LEADER]: Emulating election...
I20260321 08:56:57.882293 24578 raft_consensus.cc:3055] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 08:56:57.882510 24578 raft_consensus.cc:740] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bfcb88d0491b43828d68afbfd9b84979, State: Running, Role: LEADER
I20260321 08:56:57.882970 24578 consensus_queue.cc:260] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcb88d0491b43828d68afbfd9b84979" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38427 } } peers { permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 } }
I20260321 08:56:57.883590 24578 raft_consensus.cc:3060] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:56:57.887599 24578 raft_consensus.cc:697] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 3 LEADER]: Becoming Leader. State: Replica: bfcb88d0491b43828d68afbfd9b84979, State: Running, Role: LEADER
I20260321 08:56:57.888144 24578 consensus_queue.cc:237] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfcb88d0491b43828d68afbfd9b84979" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38427 } } peers { permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 } }
I20260321 08:56:57.889369 24578 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260321 08:56:57.892710 24578 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260321 08:56:57.996367 24578 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260321 08:56:58.200431 24578 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260321 08:56:58.492942 25345 raft_consensus.cc:3060] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881 [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:56:58.497238 25345 raft_consensus.cc:1275] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881 [term 3 FOLLOWER]: Refusing update from remote peer bfcb88d0491b43828d68afbfd9b84979: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 08:56:58.498368 25402 consensus_queue.cc:1048] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a51d601c2c949eea9e1ae1e67a33881" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 08:56:58.507390 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:56:58.528772 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:56:58.529368 24578 tablet_replica.cc:333] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979: stopping tablet replica
I20260321 08:56:58.529795 24578 raft_consensus.cc:2243] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 3 LEADER]: Raft consensus shutting down.
I20260321 08:56:58.530285 24578 raft_consensus.cc:2272] T b0e7be03a7a7457ca2c28feabfb00974 P bfcb88d0491b43828d68afbfd9b84979 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 08:56:58.548396 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:56:58.556983 24578 tablet_server.cc:179] TabletServer@127.24.0.130:0 shutting down...
I20260321 08:56:58.572984 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:56:58.573629 24578 tablet_replica.cc:333] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881: stopping tablet replica
I20260321 08:56:58.574093 24578 raft_consensus.cc:2243] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260321 08:56:58.574465 24578 raft_consensus.cc:2272] T b0e7be03a7a7457ca2c28feabfb00974 P 1a51d601c2c949eea9e1ae1e67a33881 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 08:56:58.592310 24578 tablet_server.cc:196] TabletServer@127.24.0.130:0 shutdown complete.
I20260321 08:56:58.599953 24578 master.cc:562] Master@127.24.0.190:41417 shutting down...
I20260321 08:56:58.676416 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:56:58.676940 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:56:58.677215 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 45d67c348b164ec6b30da338ddb01fa9: stopping tablet replica
I20260321 08:56:58.693725 24578 master.cc:584] Master@127.24.0.190:41417 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2579 ms)
[----------] 3 tests from LeadershipChangeReportingTest (10995 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20260321 08:56:58.722798 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:34197
I20260321 08:56:58.723995 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:58.729038 25411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:58.729676 25412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:58.731755 24578 server_base.cc:1061] running on GCE node
W20260321 08:56:58.731853 25414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:58.732609 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:58.732806 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:58.732949 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083418732929 us; error 0 us; skew 500 ppm
I20260321 08:56:58.733434 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:58.735579 24578 webserver.cc:492] Webserver started at http://127.24.0.190:32963/ using document root <none> and password file <none>
I20260321 08:56:58.736130 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:58.736302 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:58.736544 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:58.737574 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "3803cbd93a78401faab8f529e33d1a9d"
format_stamp: "Formatted at 2026-03-21 08:56:58 on dist-test-slave-p54g"
I20260321 08:56:58.741693 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260321 08:56:58.744751 25419 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:58.745375 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:56:58.745621 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "3803cbd93a78401faab8f529e33d1a9d"
format_stamp: "Formatted at 2026-03-21 08:56:58 on dist-test-slave-p54g"
I20260321 08:56:58.745889 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:58.754854 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:58.755962 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:58.791241 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:34197
I20260321 08:56:58.791324 25470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:34197 every 8 connection(s)
I20260321 08:56:58.794906 25471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:58.804064 25471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d: Bootstrap starting.
I20260321 08:56:58.808202 25471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:58.811866 25471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d: No bootstrap required, opened a new log
I20260321 08:56:58.813720 25471 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3803cbd93a78401faab8f529e33d1a9d" member_type: VOTER }
I20260321 08:56:58.814075 25471 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:56:58.814249 25471 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3803cbd93a78401faab8f529e33d1a9d, State: Initialized, Role: FOLLOWER
I20260321 08:56:58.814725 25471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3803cbd93a78401faab8f529e33d1a9d" member_type: VOTER }
I20260321 08:56:58.815120 25471 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:56:58.815306 25471 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:56:58.815502 25471 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:58.819212 25471 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3803cbd93a78401faab8f529e33d1a9d" member_type: VOTER }
I20260321 08:56:58.819631 25471 leader_election.cc:304] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3803cbd93a78401faab8f529e33d1a9d; no voters:
I20260321 08:56:58.820719 25471 leader_election.cc:290] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:56:58.821062 25474 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:56:58.822175 25474 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 1 LEADER]: Becoming Leader. State: Replica: 3803cbd93a78401faab8f529e33d1a9d, State: Running, Role: LEADER
I20260321 08:56:58.822782 25474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3803cbd93a78401faab8f529e33d1a9d" member_type: VOTER }
I20260321 08:56:58.823410 25471 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:56:58.825274 25476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3803cbd93a78401faab8f529e33d1a9d. Latest consensus state: current_term: 1 leader_uuid: "3803cbd93a78401faab8f529e33d1a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3803cbd93a78401faab8f529e33d1a9d" member_type: VOTER } }
I20260321 08:56:58.825435 25475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3803cbd93a78401faab8f529e33d1a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3803cbd93a78401faab8f529e33d1a9d" member_type: VOTER } }
I20260321 08:56:58.825917 25476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [sys.catalog]: This master's current role is: LEADER
I20260321 08:56:58.825965 25475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [sys.catalog]: This master's current role is: LEADER
I20260321 08:56:58.832513 25479 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:56:58.836831 25479 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:56:58.837893 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:56:58.844573 25479 catalog_manager.cc:1357] Generated new cluster ID: d655fab262bc414e97dc8fd5a6e4a4f9
I20260321 08:56:58.844816 25479 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:56:58.862993 25479 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:56:58.864112 25479 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:56:58.875195 25479 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d: Generated new TSK 0
I20260321 08:56:58.875716 25479 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:56:58.903955 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:58.909366 25492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:58.910264 25493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:58.912740 24578 server_base.cc:1061] running on GCE node
W20260321 08:56:58.912948 25495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:58.913827 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:58.914036 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:58.914186 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083418914169 us; error 0 us; skew 500 ppm
I20260321 08:56:58.914654 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:58.916751 24578 webserver.cc:492] Webserver started at http://127.24.0.129:43665/ using document root <none> and password file <none>
I20260321 08:56:58.917181 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:58.917346 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:58.917577 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:58.918619 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "3fbd7b4f131c4fd19f988992f836fcbc"
format_stamp: "Formatted at 2026-03-21 08:56:58 on dist-test-slave-p54g"
I20260321 08:56:58.922802 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260321 08:56:58.925813 25500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:58.926458 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260321 08:56:58.926695 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "3fbd7b4f131c4fd19f988992f836fcbc"
format_stamp: "Formatted at 2026-03-21 08:56:58 on dist-test-slave-p54g"
I20260321 08:56:58.926978 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:58.939458 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:58.940474 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:58.941876 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:56:58.944077 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:56:58.944257 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:58.944463 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:56:58.944617 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:58.980729 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:46269
I20260321 08:56:58.980819 25562 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:46269 every 8 connection(s)
I20260321 08:56:58.984732 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:58.991472 25567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:58.993332 25568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:58.995699 25570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:58.997030 24578 server_base.cc:1061] running on GCE node
I20260321 08:56:58.997134 25563 heartbeater.cc:344] Connected to a master server at 127.24.0.190:34197
I20260321 08:56:58.997539 25563 heartbeater.cc:461] Registering TS with master...
I20260321 08:56:58.997787 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:56:58.997982 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:58.998121 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083418998104 us; error 0 us; skew 500 ppm
I20260321 08:56:58.998183 25563 heartbeater.cc:507] Master 127.24.0.190:34197 requested a full tablet report, sending...
I20260321 08:56:58.998628 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:59.000186 25436 ts_manager.cc:194] Registered new tserver with Master: 3fbd7b4f131c4fd19f988992f836fcbc (127.24.0.129:46269)
I20260321 08:56:59.001574 24578 webserver.cc:492] Webserver started at http://127.24.0.130:33061/ using document root <none> and password file <none>
I20260321 08:56:59.002164 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:59.002385 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:59.002405 25436 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46710
I20260321 08:56:59.002779 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:59.004179 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-1-root/instance:
uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d"
format_stamp: "Formatted at 2026-03-21 08:56:58 on dist-test-slave-p54g"
I20260321 08:56:59.008441 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260321 08:56:59.012699 25575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:59.013550 24578 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260321 08:56:59.013868 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-1-root
uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d"
format_stamp: "Formatted at 2026-03-21 08:56:58 on dist-test-slave-p54g"
I20260321 08:56:59.014195 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:59.025243 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:59.026607 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:59.028052 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:56:59.030089 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:56:59.030275 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:59.030474 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:56:59.030614 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:59.066663 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.130:41207
I20260321 08:56:59.066746 25637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.130:41207 every 8 connection(s)
I20260321 08:56:59.070953 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:56:59.076699 25641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:59.077761 25642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:56:59.079928 25645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:56:59.080237 24578 server_base.cc:1061] running on GCE node
I20260321 08:56:59.080839 25638 heartbeater.cc:344] Connected to a master server at 127.24.0.190:34197
I20260321 08:56:59.081005 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260321 08:56:59.081146 25638 heartbeater.cc:461] Registering TS with master...
W20260321 08:56:59.081248 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:56:59.081445 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083419081428 us; error 0 us; skew 500 ppm
I20260321 08:56:59.081841 25638 heartbeater.cc:507] Master 127.24.0.190:34197 requested a full tablet report, sending...
I20260321 08:56:59.081969 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:56:59.083491 25436 ts_manager.cc:194] Registered new tserver with Master: e2a1d9eb8cca4b8dbcd8ec21c1379f0d (127.24.0.130:41207)
I20260321 08:56:59.084427 24578 webserver.cc:492] Webserver started at http://127.24.0.131:34389/ using document root <none> and password file <none>
I20260321 08:56:59.084877 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:56:59.084857 25436 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46722
I20260321 08:56:59.085156 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:56:59.085403 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:56:59.086452 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-2-root/instance:
uuid: "3a1fe58b731541ef8bc320d38f173736"
format_stamp: "Formatted at 2026-03-21 08:56:59 on dist-test-slave-p54g"
I20260321 08:56:59.090399 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260321 08:56:59.093360 25649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:59.094017 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260321 08:56:59.094260 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-2-root
uuid: "3a1fe58b731541ef8bc320d38f173736"
format_stamp: "Formatted at 2026-03-21 08:56:59 on dist-test-slave-p54g"
I20260321 08:56:59.094506 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1774083407564530-24578-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:56:59.118876 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:56:59.120047 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:56:59.121361 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:56:59.123333 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:56:59.123519 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:59.123730 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:56:59.123893 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:56:59.186302 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.131:43389
I20260321 08:56:59.186384 25711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.131:43389 every 8 connection(s)
I20260321 08:56:59.199556 25712 heartbeater.cc:344] Connected to a master server at 127.24.0.190:34197
I20260321 08:56:59.199868 25712 heartbeater.cc:461] Registering TS with master...
I20260321 08:56:59.200562 25712 heartbeater.cc:507] Master 127.24.0.190:34197 requested a full tablet report, sending...
I20260321 08:56:59.202168 25436 ts_manager.cc:194] Registered new tserver with Master: 3a1fe58b731541ef8bc320d38f173736 (127.24.0.131:43389)
I20260321 08:56:59.202524 24578 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012062451s
I20260321 08:56:59.203346 25436 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46732
I20260321 08:56:59.226146 25436 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260321 08:56:59.227620 25436 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260321 08:56:59.269555 25528 tablet_service.cc:1511] Processing CreateTablet for tablet 61cc8e4a03564e56b378b47a43b33432 (DEFAULT_TABLE table=test-table [id=a3df5f6314584c96a57c3ac3e32ad9cd]), partition=
I20260321 08:56:59.270751 25528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61cc8e4a03564e56b378b47a43b33432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:59.279606 25603 tablet_service.cc:1511] Processing CreateTablet for tablet 61cc8e4a03564e56b378b47a43b33432 (DEFAULT_TABLE table=test-table [id=a3df5f6314584c96a57c3ac3e32ad9cd]), partition=
I20260321 08:56:59.280434 25603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61cc8e4a03564e56b378b47a43b33432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:59.280663 25677 tablet_service.cc:1511] Processing CreateTablet for tablet 61cc8e4a03564e56b378b47a43b33432 (DEFAULT_TABLE table=test-table [id=a3df5f6314584c96a57c3ac3e32ad9cd]), partition=
I20260321 08:56:59.281636 25677 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61cc8e4a03564e56b378b47a43b33432. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:56:59.287549 25727 tablet_bootstrap.cc:492] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc: Bootstrap starting.
I20260321 08:56:59.294340 25727 tablet_bootstrap.cc:654] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:59.294775 25728 tablet_bootstrap.cc:492] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d: Bootstrap starting.
I20260321 08:56:59.296058 25729 tablet_bootstrap.cc:492] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736: Bootstrap starting.
I20260321 08:56:59.299752 25727 tablet_bootstrap.cc:492] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc: No bootstrap required, opened a new log
I20260321 08:56:59.300143 25727 ts_tablet_manager.cc:1403] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260321 08:56:59.302045 25728 tablet_bootstrap.cc:654] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:59.302147 25727 raft_consensus.cc:359] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 } } peers { permanent_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41207 } } peers { permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 } }
I20260321 08:56:59.302853 25729 tablet_bootstrap.cc:654] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736: Neither blocks nor log segments found. Creating new log.
I20260321 08:56:59.302805 25727 raft_consensus.cc:385] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:56:59.303403 25727 raft_consensus.cc:740] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fbd7b4f131c4fd19f988992f836fcbc, State: Initialized, Role: FOLLOWER
I20260321 08:56:59.304293 25727 consensus_queue.cc:260] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 } } peers { permanent_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41207 } } peers { permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 } }
I20260321 08:56:59.309001 25563 heartbeater.cc:499] Master 127.24.0.190:34197 was elected leader, sending a full tablet report...
I20260321 08:56:59.309223 25728 tablet_bootstrap.cc:492] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d: No bootstrap required, opened a new log
I20260321 08:56:59.309193 25727 ts_tablet_manager.cc:1434] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260321 08:56:59.309556 25729 tablet_bootstrap.cc:492] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736: No bootstrap required, opened a new log
I20260321 08:56:59.309751 25728 ts_tablet_manager.cc:1403] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260321 08:56:59.310160 25729 ts_tablet_manager.cc:1403] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260321 08:56:59.312470 25729 raft_consensus.cc:359] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 } } peers { permanent_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41207 } } peers { permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 } }
I20260321 08:56:59.312390 25728 raft_consensus.cc:359] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 } } peers { permanent_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41207 } } peers { permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 } }
I20260321 08:56:59.313046 25729 raft_consensus.cc:385] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:56:59.313197 25728 raft_consensus.cc:385] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:56:59.313364 25729 raft_consensus.cc:740] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1fe58b731541ef8bc320d38f173736, State: Initialized, Role: FOLLOWER
I20260321 08:56:59.313571 25728 raft_consensus.cc:740] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a1d9eb8cca4b8dbcd8ec21c1379f0d, State: Initialized, Role: FOLLOWER
I20260321 08:56:59.314023 25729 consensus_queue.cc:260] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 } } peers { permanent_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41207 } } peers { permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 } }
I20260321 08:56:59.314410 25728 consensus_queue.cc:260] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 } } peers { permanent_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41207 } } peers { permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 } }
I20260321 08:56:59.316797 25729 ts_tablet_manager.cc:1434] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260321 08:56:59.317456 25712 heartbeater.cc:499] Master 127.24.0.190:34197 was elected leader, sending a full tablet report...
I20260321 08:56:59.321971 25638 heartbeater.cc:499] Master 127.24.0.190:34197 was elected leader, sending a full tablet report...
I20260321 08:56:59.322434 25728 ts_tablet_manager.cc:1434] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
I20260321 08:56:59.379787 25735 raft_consensus.cc:493] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:56:59.380270 25735 raft_consensus.cc:515] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 } } peers { permanent_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41207 } } peers { permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 } }
I20260321 08:56:59.383507 25735 leader_election.cc:290] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fbd7b4f131c4fd19f988992f836fcbc (127.24.0.129:46269), 3a1fe58b731541ef8bc320d38f173736 (127.24.0.131:43389)
I20260321 08:56:59.390802 25538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61cc8e4a03564e56b378b47a43b33432" candidate_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" is_pre_election: true
I20260321 08:56:59.391621 25538 raft_consensus.cc:2468] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a1d9eb8cca4b8dbcd8ec21c1379f0d in term 0.
I20260321 08:56:59.392061 25687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61cc8e4a03564e56b378b47a43b33432" candidate_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1fe58b731541ef8bc320d38f173736" is_pre_election: true
I20260321 08:56:59.392647 25687 raft_consensus.cc:2468] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a1d9eb8cca4b8dbcd8ec21c1379f0d in term 0.
I20260321 08:56:59.392706 25579 leader_election.cc:304] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fbd7b4f131c4fd19f988992f836fcbc, e2a1d9eb8cca4b8dbcd8ec21c1379f0d; no voters:
I20260321 08:56:59.393504 25735 raft_consensus.cc:2804] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:56:59.393765 25735 raft_consensus.cc:493] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:56:59.394006 25735 raft_consensus.cc:3060] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:59.398500 25735 raft_consensus.cc:515] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 } } peers { permanent_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41207 } } peers { permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 } }
I20260321 08:56:59.400004 25735 leader_election.cc:290] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [CANDIDATE]: Term 1 election: Requested vote from peers 3fbd7b4f131c4fd19f988992f836fcbc (127.24.0.129:46269), 3a1fe58b731541ef8bc320d38f173736 (127.24.0.131:43389)
I20260321 08:56:59.401000 25687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61cc8e4a03564e56b378b47a43b33432" candidate_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a1fe58b731541ef8bc320d38f173736"
I20260321 08:56:59.401041 25538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61cc8e4a03564e56b378b47a43b33432" candidate_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fbd7b4f131c4fd19f988992f836fcbc"
I20260321 08:56:59.401513 25687 raft_consensus.cc:3060] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:59.401543 25538 raft_consensus.cc:3060] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:56:59.406286 25687 raft_consensus.cc:2468] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a1d9eb8cca4b8dbcd8ec21c1379f0d in term 1.
I20260321 08:56:59.406257 25538 raft_consensus.cc:2468] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a1d9eb8cca4b8dbcd8ec21c1379f0d in term 1.
I20260321 08:56:59.407409 25576 leader_election.cc:304] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a1fe58b731541ef8bc320d38f173736, e2a1d9eb8cca4b8dbcd8ec21c1379f0d; no voters:
I20260321 08:56:59.408167 25735 raft_consensus.cc:2804] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:56:59.409221 25735 raft_consensus.cc:697] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 1 LEADER]: Becoming Leader. State: Replica: e2a1d9eb8cca4b8dbcd8ec21c1379f0d, State: Running, Role: LEADER
I20260321 08:56:59.409981 25735 consensus_queue.cc:237] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 } } peers { permanent_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41207 } } peers { permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 } }
I20260321 08:56:59.416236 25435 catalog_manager.cc:5671] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d reported cstate change: term changed from 0 to 1, leader changed from <none> to e2a1d9eb8cca4b8dbcd8ec21c1379f0d (127.24.0.130). New cstate: current_term: 1 leader_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2a1d9eb8cca4b8dbcd8ec21c1379f0d" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 } health_report { overall_health: UNKNOWN } } }
I20260321 08:56:59.889051 25735 consensus_queue.cc:1048] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fbd7b4f131c4fd19f988992f836fcbc" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 46269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:56:59.899076 25743 consensus_queue.cc:1048] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a1fe58b731541ef8bc320d38f173736" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:00.499961 24578 tablet_replica.cc:333] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc: stopping tablet replica
I20260321 08:57:00.500460 24578 raft_consensus.cc:2243] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:00.500803 24578 raft_consensus.cc:2272] T 61cc8e4a03564e56b378b47a43b33432 P 3fbd7b4f131c4fd19f988992f836fcbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:00.800290 25579 consensus_queue.cc:579] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [LEADER]: The tablet replica hosted on peer 3fbd7b4f131c4fd19f988992f836fcbc has failed
W20260321 08:57:00.800700 25579 consensus_peers.cc:597] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d -> Peer 3fbd7b4f131c4fd19f988992f836fcbc (127.24.0.129:46269): Couldn't send request to peer 3fbd7b4f131c4fd19f988992f836fcbc. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:01.023710 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:57:01.038735 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:01.054891 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:57:01.061908 24578 tablet_server.cc:179] TabletServer@127.24.0.130:0 shutting down...
I20260321 08:57:01.075313 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:01.075915 24578 tablet_replica.cc:333] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d: stopping tablet replica
I20260321 08:57:01.076464 24578 raft_consensus.cc:2243] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:01.077150 24578 raft_consensus.cc:2272] T 61cc8e4a03564e56b378b47a43b33432 P e2a1d9eb8cca4b8dbcd8ec21c1379f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:01.094158 24578 tablet_server.cc:196] TabletServer@127.24.0.130:0 shutdown complete.
I20260321 08:57:01.101506 24578 tablet_server.cc:179] TabletServer@127.24.0.131:0 shutting down...
I20260321 08:57:01.115250 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:01.115844 24578 tablet_replica.cc:333] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736: stopping tablet replica
I20260321 08:57:01.116339 24578 raft_consensus.cc:2243] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:01.116747 24578 raft_consensus.cc:2272] T 61cc8e4a03564e56b378b47a43b33432 P 3a1fe58b731541ef8bc320d38f173736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:01.133680 24578 tablet_server.cc:196] TabletServer@127.24.0.131:0 shutdown complete.
I20260321 08:57:01.141005 24578 master.cc:562] Master@127.24.0.190:34197 shutting down...
I20260321 08:57:01.153115 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:01.153546 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:01.153779 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3803cbd93a78401faab8f529e33d1a9d: stopping tablet replica
I20260321 08:57:01.169543 24578 master.cc:584] Master@127.24.0.190:34197 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (2469 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20260321 08:57:01.191465 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:37491
I20260321 08:57:01.192451 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:01.196756 25752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:01.197973 25753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:01.199210 25755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:01.200181 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:01.200819 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:01.201004 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:01.201144 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083421201127 us; error 0 us; skew 500 ppm
I20260321 08:57:01.201653 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:01.203723 24578 webserver.cc:492] Webserver started at http://127.24.0.190:44643/ using document root <none> and password file <none>
I20260321 08:57:01.204193 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:01.204349 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:01.204543 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:01.205482 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "b64f037dc77541fd93e2b56e6b40e891"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.209416 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260321 08:57:01.212383 25760 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:01.213076 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260321 08:57:01.213340 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "b64f037dc77541fd93e2b56e6b40e891"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.213599 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:01.242528 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:01.243506 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:01.276921 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:37491
I20260321 08:57:01.277026 25811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:37491 every 8 connection(s)
I20260321 08:57:01.280395 25812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:01.289198 25812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891: Bootstrap starting.
I20260321 08:57:01.293063 25812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:01.296527 25812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891: No bootstrap required, opened a new log
I20260321 08:57:01.298182 25812 raft_consensus.cc:359] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64f037dc77541fd93e2b56e6b40e891" member_type: VOTER }
I20260321 08:57:01.298518 25812 raft_consensus.cc:385] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:01.298746 25812 raft_consensus.cc:740] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64f037dc77541fd93e2b56e6b40e891, State: Initialized, Role: FOLLOWER
I20260321 08:57:01.299273 25812 consensus_queue.cc:260] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64f037dc77541fd93e2b56e6b40e891" member_type: VOTER }
I20260321 08:57:01.299690 25812 raft_consensus.cc:399] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:01.299926 25812 raft_consensus.cc:493] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:01.300166 25812 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:01.303918 25812 raft_consensus.cc:515] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64f037dc77541fd93e2b56e6b40e891" member_type: VOTER }
I20260321 08:57:01.304345 25812 leader_election.cc:304] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b64f037dc77541fd93e2b56e6b40e891; no voters:
I20260321 08:57:01.305323 25812 leader_election.cc:290] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:01.305617 25815 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:01.306777 25815 raft_consensus.cc:697] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 1 LEADER]: Becoming Leader. State: Replica: b64f037dc77541fd93e2b56e6b40e891, State: Running, Role: LEADER
I20260321 08:57:01.307389 25815 consensus_queue.cc:237] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64f037dc77541fd93e2b56e6b40e891" member_type: VOTER }
I20260321 08:57:01.307916 25812 sys_catalog.cc:565] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:01.310022 25816 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b64f037dc77541fd93e2b56e6b40e891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64f037dc77541fd93e2b56e6b40e891" member_type: VOTER } }
I20260321 08:57:01.310081 25817 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b64f037dc77541fd93e2b56e6b40e891. Latest consensus state: current_term: 1 leader_uuid: "b64f037dc77541fd93e2b56e6b40e891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64f037dc77541fd93e2b56e6b40e891" member_type: VOTER } }
I20260321 08:57:01.310774 25817 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:01.311786 25816 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:01.316131 25821 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:01.322567 25821 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:01.324474 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:57:01.330485 25821 catalog_manager.cc:1357] Generated new cluster ID: 871115de0ecc4e92a1ef08b56b50a08e
I20260321 08:57:01.330744 25821 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:01.360742 25821 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:57:01.361924 25821 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:01.380739 25821 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891: Generated new TSK 0
I20260321 08:57:01.381347 25821 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:01.390364 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:01.396164 25833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:01.396804 25834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:01.398249 25836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:01.398931 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:01.399636 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:01.399818 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:01.400089 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083421400065 us; error 0 us; skew 500 ppm
I20260321 08:57:01.400589 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:01.402823 24578 webserver.cc:492] Webserver started at http://127.24.0.129:38199/ using document root <none> and password file <none>
I20260321 08:57:01.403256 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:01.403402 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:01.403599 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:01.404645 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "20e246978b264c8b8f807072a05465d5"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.408872 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260321 08:57:01.412045 25841 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:01.412832 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260321 08:57:01.413090 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "20e246978b264c8b8f807072a05465d5"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.413326 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:01.434562 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:01.435611 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:01.436968 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:01.439035 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:01.439200 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:01.439424 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:01.439558 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:01.474103 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:42393
I20260321 08:57:01.474198 25903 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:42393 every 8 connection(s)
I20260321 08:57:01.490480 25904 heartbeater.cc:344] Connected to a master server at 127.24.0.190:37491
I20260321 08:57:01.490818 25904 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:01.491475 25904 heartbeater.cc:507] Master 127.24.0.190:37491 requested a full tablet report, sending...
I20260321 08:57:01.493357 25777 ts_manager.cc:194] Registered new tserver with Master: 20e246978b264c8b8f807072a05465d5 (127.24.0.129:42393)
I20260321 08:57:01.493417 24578 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.01632885s
I20260321 08:57:01.495276 25777 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49266
I20260321 08:57:01.512888 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:57:01.527225 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:01.542062 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:57:01.548913 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260321 08:57:01.551386 24578 tablet_server.cc:94] deduplicated master addresses: 127.24.0.190:37491
W20260321 08:57:01.554155 25914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:01.555545 25915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:01.555774 25917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:01.556186 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:01.556900 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:01.557080 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:01.557220 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083421557203 us; error 0 us; skew 500 ppm
I20260321 08:57:01.557705 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:01.559690 24578 webserver.cc:492] Webserver started at http://127.24.0.129:38199/ using document root <none> and password file <none>
I20260321 08:57:01.560132 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:01.560283 24578 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260321 08:57:01.563377 24578 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:57:01.565804 25922 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:01.566484 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260321 08:57:01.566747 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "20e246978b264c8b8f807072a05465d5"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.567010 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:01.587364 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:01.588337 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:01.589732 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:01.591755 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:01.591952 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:01.592165 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:01.592325 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:01.627737 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:42393
I20260321 08:57:01.627818 25984 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:42393 every 8 connection(s)
I20260321 08:57:01.634195 24578 tablet_server.cc:179] TabletServer@127.24.0.129:42393 shutting down...
I20260321 08:57:01.645399 25985 heartbeater.cc:344] Connected to a master server at 127.24.0.190:37491
I20260321 08:57:01.645741 25985 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:01.646581 25985 heartbeater.cc:507] Master 127.24.0.190:37491 requested a full tablet report, sending...
I20260321 08:57:01.649762 25777 ts_manager.cc:194] Re-registered known tserver with Master: 20e246978b264c8b8f807072a05465d5 (127.24.0.129:42393)
I20260321 08:57:01.651978 25777 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49278
I20260321 08:57:01.654242 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:01.668864 24578 tablet_server.cc:196] TabletServer@127.24.0.129:42393 shutdown complete.
I20260321 08:57:01.675326 24578 master.cc:562] Master@127.24.0.190:37491 shutting down...
I20260321 08:57:01.687417 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:01.687906 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:01.688283 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P b64f037dc77541fd93e2b56e6b40e891: stopping tablet replica
I20260321 08:57:01.705358 24578 master.cc:584] Master@127.24.0.190:37491 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (533 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20260321 08:57:01.725585 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:45875,127.24.0.189:38105,127.24.0.188:45371
I20260321 08:57:01.726797 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:01.731209 25992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:01.732252 25993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:01.733668 25995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:01.734522 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:01.735108 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:01.735266 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:01.735375 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083421735365 us; error 0 us; skew 500 ppm
I20260321 08:57:01.735811 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:01.737938 24578 webserver.cc:492] Webserver started at http://127.24.0.190:38293/ using document root <none> and password file <none>
I20260321 08:57:01.738322 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:01.738458 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:01.738664 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:01.739588 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "42e553cc2497492a9aa4dd5cb1d3ae74"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.743597 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260321 08:57:01.746749 26000 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:01.747476 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260321 08:57:01.747743 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "42e553cc2497492a9aa4dd5cb1d3ae74"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.748055 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:01.772163 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:01.773092 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:01.805549 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:45875
I20260321 08:57:01.805649 26051 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:45875 every 8 connection(s)
I20260321 08:57:01.808992 26052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:01.809115 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:01.813881 26054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:01.815864 26052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } has no permanent_uuid. Determining permanent_uuid...
W20260321 08:57:01.816493 26055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:01.817288 26057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:01.817795 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:01.818514 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:01.818699 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:01.818858 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083421818842 us; error 0 us; skew 500 ppm
I20260321 08:57:01.819334 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:01.821866 24578 webserver.cc:492] Webserver started at http://127.24.0.189:35349/ using document root <none> and password file <none>
I20260321 08:57:01.822376 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:01.822531 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:01.822774 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:01.823897 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-1-root/instance:
uuid: "5c9e21016d934beab3beb647aa48006b"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.828768 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260321 08:57:01.829056 26052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } has no permanent_uuid. Determining permanent_uuid...
W20260321 08:57:01.831230 26004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111)
I20260321 08:57:01.832345 26064 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:01.833205 24578 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260321 08:57:01.833449 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-1-root
uuid: "5c9e21016d934beab3beb647aa48006b"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.833779 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260321 08:57:01.834518 26052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.0.189:38105: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111)
I20260321 08:57:01.865732 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:01.866719 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:01.886071 26052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } attempt: 1
W20260321 08:57:01.891069 26052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.0.189:38105: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111)
I20260321 08:57:01.901510 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.189:38105
I20260321 08:57:01.901611 26116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.189:38105 every 8 connection(s)
I20260321 08:57:01.904938 26117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:01.905061 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:01.910348 26119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:01.911906 26120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:01.912501 26117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } has no permanent_uuid. Determining permanent_uuid...
I20260321 08:57:01.914038 24578 server_base.cc:1061] running on GCE node
W20260321 08:57:01.915751 26122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:01.916563 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:01.916771 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:01.916932 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083421916914 us; error 0 us; skew 500 ppm
I20260321 08:57:01.917515 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:01.920034 24578 webserver.cc:492] Webserver started at http://127.24.0.188:43489/ using document root <none> and password file <none>
I20260321 08:57:01.920452 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:01.920639 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:01.920899 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:01.921859 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-2-root/instance:
uuid: "9af081622eaa4661a29feb3bf5d18cc8"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.923424 26117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } has no permanent_uuid. Determining permanent_uuid...
I20260321 08:57:01.926829 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260321 08:57:01.930732 26129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:01.931613 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260321 08:57:01.931870 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-2-root
uuid: "9af081622eaa4661a29feb3bf5d18cc8"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:01.932159 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:01.934131 26117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
W20260321 08:57:01.938969 26117 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.0.188:45371: Network error: Client connection negotiation failed: client connection to 127.24.0.188:45371: connect: Connection refused (error 111)
I20260321 08:57:01.952549 26052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } attempt: 2
I20260321 08:57:01.957401 26117 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } attempt: 1
W20260321 08:57:01.961649 26117 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.0.188:45371: Network error: Client connection negotiation failed: client connection to 127.24.0.188:45371: connect: Connection refused (error 111)
I20260321 08:57:01.962822 26052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
I20260321 08:57:01.966225 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20260321 08:57:01.967074 26052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.0.188:45371: Network error: Client connection negotiation failed: client connection to 127.24.0.188:45371: connect: Connection refused (error 111)
I20260321 08:57:01.967391 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:01.991539 26052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } attempt: 1
W20260321 08:57:01.995986 26052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.24.0.188:45371: Network error: Client connection negotiation failed: client connection to 127.24.0.188:45371: connect: Connection refused (error 111)
I20260321 08:57:02.002074 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.188:45371
I20260321 08:57:02.002154 26181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.188:45371 every 8 connection(s)
I20260321 08:57:02.004829 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:57:02.005460 26182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:02.010798 26182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } has no permanent_uuid. Determining permanent_uuid...
I20260321 08:57:02.016063 26117 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } attempt: 2
I20260321 08:57:02.021437 26182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } has no permanent_uuid. Determining permanent_uuid...
I20260321 08:57:02.030936 26182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
I20260321 08:57:02.033245 26117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: Bootstrap starting.
I20260321 08:57:02.037451 26117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:02.041503 26117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: No bootstrap required, opened a new log
I20260321 08:57:02.043267 26117 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:02.043736 26117 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:02.044009 26117 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9e21016d934beab3beb647aa48006b, State: Initialized, Role: FOLLOWER
I20260321 08:57:02.044497 26182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8: Bootstrap starting.
I20260321 08:57:02.044564 26117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:02.046372 26187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:02.047072 26187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:02.047416 26052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } attempt: 2
I20260321 08:57:02.047811 26117 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:02.048880 26182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:02.056365 26182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8: No bootstrap required, opened a new log
I20260321 08:57:02.058509 26182 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:02.059190 26182 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:02.059520 26182 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af081622eaa4661a29feb3bf5d18cc8, State: Initialized, Role: FOLLOWER
I20260321 08:57:02.060264 26182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:02.064216 26052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74: Bootstrap starting.
W20260321 08:57:02.067117 26200 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260321 08:57:02.067499 26200 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260321 08:57:02.068486 26052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:02.068329 26199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:02.069065 26199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:02.069458 26182 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:02.072726 26052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74: No bootstrap required, opened a new log
I20260321 08:57:02.075021 26052 raft_consensus.cc:359] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:02.075467 26052 raft_consensus.cc:385] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:02.075695 26052 raft_consensus.cc:740] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e553cc2497492a9aa4dd5cb1d3ae74, State: Initialized, Role: FOLLOWER
I20260321 08:57:02.076248 26052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:02.078366 26199 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:02.078789 26199 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:02.079243 26052 sys_catalog.cc:565] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:02.080857 26199 leader_election.cc:290] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105)
I20260321 08:57:02.081552 26027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
I20260321 08:57:02.082183 26027 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9af081622eaa4661a29feb3bf5d18cc8 in term 0.
I20260321 08:57:02.082404 26092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b" is_pre_election: true
I20260321 08:57:02.083312 26092 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9af081622eaa4661a29feb3bf5d18cc8 in term 0.
I20260321 08:57:02.083390 26130 leader_election.cc:304] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8; no voters:
I20260321 08:57:02.078034 26209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:02.085638 26199 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:57:02.085873 26209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:02.086123 26199 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:02.086560 26199 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 0 FOLLOWER]: Advancing to term 1
W20260321 08:57:02.089124 26213 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260321 08:57:02.089428 26213 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260321 08:57:02.092077 26199 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:02.093540 26199 leader_election.cc:290] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 1 election: Requested vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105)
I20260321 08:57:02.094552 26092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b"
I20260321 08:57:02.094630 26027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74"
I20260321 08:57:02.094985 26092 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:02.095368 26027 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:02.099330 26092 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9af081622eaa4661a29feb3bf5d18cc8 in term 1.
I20260321 08:57:02.100339 26134 leader_election.cc:304] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9e21016d934beab3beb647aa48006b, 9af081622eaa4661a29feb3bf5d18cc8; no voters:
I20260321 08:57:02.100982 26199 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:02.101497 26027 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9af081622eaa4661a29feb3bf5d18cc8 in term 1.
I20260321 08:57:02.103550 26199 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 1 LEADER]: Becoming Leader. State: Replica: 9af081622eaa4661a29feb3bf5d18cc8, State: Running, Role: LEADER
I20260321 08:57:02.104203 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20260321 08:57:02.104460 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20260321 08:57:02.104387 26199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:02.106138 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:02.107932 26225 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260321 08:57:02.108273 26225 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
E20260321 08:57:02.108286 24578 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260321 08:57:02.109175 26224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af081622eaa4661a29feb3bf5d18cc8. Latest consensus state: current_term: 1 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:02.109870 26224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:02.111482 26227 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
W20260321 08:57:02.113422 26228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:02.114667 26229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:02.115492 26231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:02.116401 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:02.117336 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:02.117588 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:02.117800 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083422117783 us; error 0 us; skew 500 ppm
I20260321 08:57:02.118431 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:02.118707 26227 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:02.121184 24578 webserver.cc:492] Webserver started at http://127.24.0.129:38175/ using document root <none> and password file <none>
I20260321 08:57:02.121798 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:02.122022 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:02.122334 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:02.123562 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "de2947147ecb4d6eafa4aa039126f25b"
format_stamp: "Formatted at 2026-03-21 08:57:02 on dist-test-slave-p54g"
I20260321 08:57:02.127010 26027 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 1 FOLLOWER]: Refusing update from remote peer 9af081622eaa4661a29feb3bf5d18cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 08:57:02.126991 26092 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 1 FOLLOWER]: Refusing update from remote peer 9af081622eaa4661a29feb3bf5d18cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 08:57:02.128441 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260321 08:57:02.128554 26199 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:02.129191 26224 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:02.133039 26237 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:02.134058 24578 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260321 08:57:02.134562 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "de2947147ecb4d6eafa4aa039126f25b"
format_stamp: "Formatted at 2026-03-21 08:57:02 on dist-test-slave-p54g"
I20260321 08:57:02.134974 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:02.136896 26209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af081622eaa4661a29feb3bf5d18cc8. Latest consensus state: current_term: 1 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:02.137540 26209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:02.143187 26199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:02.143951 26199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:02.146772 26227 catalog_manager.cc:1357] Generated new cluster ID: 9774d2365c48473a92ff616a29d5cfc9
I20260321 08:57:02.147070 26227 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:02.153124 26209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:02.153770 26209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:02.161067 26187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af081622eaa4661a29feb3bf5d18cc8. Latest consensus state: current_term: 1 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:02.161568 26187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:02.164583 26224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:02.165290 26224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:02.173833 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:02.175531 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:02.177222 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:02.177770 26187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:02.178445 26187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:02.180405 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:02.180809 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:02.181103 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:02.181305 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:02.191246 26227 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:57:02.193008 26227 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:02.217610 26227 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8: Generated new TSK 0
I20260321 08:57:02.218155 26227 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:02.241354 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:38365
I20260321 08:57:02.241429 26308 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:38365 every 8 connection(s)
I20260321 08:57:02.248535 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260321 08:57:02.250814 24578 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260321 08:57:02.260589 26316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:02.261719 26309 heartbeater.cc:344] Connected to a master server at 127.24.0.188:45371
I20260321 08:57:02.261893 26310 heartbeater.cc:344] Connected to a master server at 127.24.0.189:38105
I20260321 08:57:02.262180 26309 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:02.263149 26309 heartbeater.cc:507] Master 127.24.0.188:45371 requested a full tablet report, sending...
I20260321 08:57:02.262249 26310 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:02.264833 26310 heartbeater.cc:507] Master 127.24.0.189:38105 requested a full tablet report, sending...
I20260321 08:57:02.265681 26147 ts_manager.cc:194] Registered new tserver with Master: de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365)
I20260321 08:57:02.267679 26082 ts_manager.cc:194] Registered new tserver with Master: de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365)
I20260321 08:57:02.268038 26147 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47132
W20260321 08:57:02.270931 26317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:02.271106 26311 heartbeater.cc:344] Connected to a master server at 127.24.0.190:45875
I20260321 08:57:02.271546 26311 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:02.272213 26311 heartbeater.cc:507] Master 127.24.0.190:45875 requested a full tablet report, sending...
I20260321 08:57:02.275060 24578 server_base.cc:1061] running on GCE node
W20260321 08:57:02.274994 26319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:02.275169 26017 ts_manager.cc:194] Registered new tserver with Master: de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365)
I20260321 08:57:02.276325 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:02.276593 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:02.276782 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083422276763 us; error 0 us; skew 500 ppm
I20260321 08:57:02.277297 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:02.279389 24578 webserver.cc:492] Webserver started at http://127.24.0.130:41965/ using document root <none> and password file <none>
I20260321 08:57:02.279819 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:02.280047 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:02.280282 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:02.281282 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-1-root/instance:
uuid: "d119b009c5744457b0555896e263fa35"
format_stamp: "Formatted at 2026-03-21 08:57:02 on dist-test-slave-p54g"
I20260321 08:57:02.285344 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260321 08:57:02.288318 26324 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:02.288976 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:57:02.289227 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-1-root
uuid: "d119b009c5744457b0555896e263fa35"
format_stamp: "Formatted at 2026-03-21 08:57:02 on dist-test-slave-p54g"
I20260321 08:57:02.289469 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:02.309587 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:02.310714 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:02.312547 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:02.315091 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:02.315287 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:02.315506 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:02.315680 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:02.370709 26309 heartbeater.cc:499] Master 127.24.0.188:45371 was elected leader, sending a full tablet report...
I20260321 08:57:02.388670 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.130:41799
I20260321 08:57:02.388761 26386 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.130:41799 every 8 connection(s)
I20260321 08:57:02.399632 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260321 08:57:02.402096 24578 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260321 08:57:02.412112 26387 heartbeater.cc:344] Connected to a master server at 127.24.0.188:45371
I20260321 08:57:02.412555 26387 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:02.413450 26387 heartbeater.cc:507] Master 127.24.0.188:45371 requested a full tablet report, sending...
I20260321 08:57:02.416435 26147 ts_manager.cc:194] Registered new tserver with Master: d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:02.418124 26147 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47146
I20260321 08:57:02.419905 26388 heartbeater.cc:344] Connected to a master server at 127.24.0.189:38105
I20260321 08:57:02.420311 26388 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:02.421217 26388 heartbeater.cc:507] Master 127.24.0.189:38105 requested a full tablet report, sending...
I20260321 08:57:02.424293 26082 ts_manager.cc:194] Registered new tserver with Master: d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
W20260321 08:57:02.425925 26394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:02.426144 24578 server_base.cc:1061] running on GCE node
W20260321 08:57:02.426386 26395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:02.426544 26389 heartbeater.cc:344] Connected to a master server at 127.24.0.190:45875
I20260321 08:57:02.426923 26389 heartbeater.cc:461] Registering TS with master...
W20260321 08:57:02.427276 26397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:02.427513 26389 heartbeater.cc:507] Master 127.24.0.190:45875 requested a full tablet report, sending...
I20260321 08:57:02.428139 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:02.428380 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:02.428542 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083422428524 us; error 0 us; skew 500 ppm
I20260321 08:57:02.429064 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:02.429629 26017 ts_manager.cc:194] Registered new tserver with Master: d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:02.431707 24578 webserver.cc:492] Webserver started at http://127.24.0.131:40181/ using document root <none> and password file <none>
I20260321 08:57:02.432210 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:02.432394 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:02.432636 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:02.433580 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-2-root/instance:
uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
format_stamp: "Formatted at 2026-03-21 08:57:02 on dist-test-slave-p54g"
I20260321 08:57:02.437955 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260321 08:57:02.441403 26402 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:02.442142 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:57:02.442381 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-2-root
uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
format_stamp: "Formatted at 2026-03-21 08:57:02 on dist-test-slave-p54g"
I20260321 08:57:02.442606 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:02.459806 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:02.461027 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:02.462199 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:02.464732 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:02.464926 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:02.465145 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:02.465298 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:02.522996 26387 heartbeater.cc:499] Master 127.24.0.188:45371 was elected leader, sending a full tablet report...
I20260321 08:57:02.538163 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.131:36993
I20260321 08:57:02.538317 26464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.131:36993 every 8 connection(s)
I20260321 08:57:02.557642 26465 heartbeater.cc:344] Connected to a master server at 127.24.0.188:45371
I20260321 08:57:02.558068 26465 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:02.558853 26465 heartbeater.cc:507] Master 127.24.0.188:45371 requested a full tablet report, sending...
I20260321 08:57:02.560555 26466 heartbeater.cc:344] Connected to a master server at 127.24.0.189:38105
I20260321 08:57:02.560923 26466 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:02.561793 26466 heartbeater.cc:507] Master 127.24.0.189:38105 requested a full tablet report, sending...
I20260321 08:57:02.564705 26082 ts_manager.cc:194] Registered new tserver with Master: dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:02.560945 26147 ts_manager.cc:194] Registered new tserver with Master: dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:02.567010 26147 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47154
I20260321 08:57:02.572165 26468 heartbeater.cc:344] Connected to a master server at 127.24.0.190:45875
I20260321 08:57:02.572480 26468 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:02.573086 26468 heartbeater.cc:507] Master 127.24.0.190:45875 requested a full tablet report, sending...
I20260321 08:57:02.575182 26017 ts_manager.cc:194] Registered new tserver with Master: dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:02.575327 24578 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005353319s
I20260321 08:57:02.637508 26147 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20260321 08:57:02.639040 26147 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260321 08:57:02.669879 26465 heartbeater.cc:499] Master 127.24.0.188:45371 was elected leader, sending a full tablet report...
I20260321 08:57:02.704239 26352 tablet_service.cc:1511] Processing CreateTablet for tablet 3effe2075d034127a1ac31b507b8119a (DEFAULT_TABLE table=test-table [id=aec8b423bf774e35beabf2b480fcfda8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:02.705201 26350 tablet_service.cc:1511] Processing CreateTablet for tablet 51dc0b437c8c49aea324853455672c6c (DEFAULT_TABLE table=test-table [id=aec8b423bf774e35beabf2b480fcfda8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:02.705704 26352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3effe2075d034127a1ac31b507b8119a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:02.704456 26351 tablet_service.cc:1511] Processing CreateTablet for tablet b4eff2c92f994bb2a318dfe6097efacb (DEFAULT_TABLE table=test-table [id=aec8b423bf774e35beabf2b480fcfda8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:02.706470 26350 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51dc0b437c8c49aea324853455672c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:02.708122 26274 tablet_service.cc:1511] Processing CreateTablet for tablet 3effe2075d034127a1ac31b507b8119a (DEFAULT_TABLE table=test-table [id=aec8b423bf774e35beabf2b480fcfda8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:02.708436 26273 tablet_service.cc:1511] Processing CreateTablet for tablet b4eff2c92f994bb2a318dfe6097efacb (DEFAULT_TABLE table=test-table [id=aec8b423bf774e35beabf2b480fcfda8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:02.709483 26274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3effe2075d034127a1ac31b507b8119a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:02.710064 26351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4eff2c92f994bb2a318dfe6097efacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:02.710101 26273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4eff2c92f994bb2a318dfe6097efacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:02.711628 26272 tablet_service.cc:1511] Processing CreateTablet for tablet 51dc0b437c8c49aea324853455672c6c (DEFAULT_TABLE table=test-table [id=aec8b423bf774e35beabf2b480fcfda8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:02.713083 26272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51dc0b437c8c49aea324853455672c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:02.724583 26430 tablet_service.cc:1511] Processing CreateTablet for tablet 3effe2075d034127a1ac31b507b8119a (DEFAULT_TABLE table=test-table [id=aec8b423bf774e35beabf2b480fcfda8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:02.725039 26429 tablet_service.cc:1511] Processing CreateTablet for tablet b4eff2c92f994bb2a318dfe6097efacb (DEFAULT_TABLE table=test-table [id=aec8b423bf774e35beabf2b480fcfda8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:02.725836 26430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3effe2075d034127a1ac31b507b8119a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:02.726547 26429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4eff2c92f994bb2a318dfe6097efacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:02.727653 26428 tablet_service.cc:1511] Processing CreateTablet for tablet 51dc0b437c8c49aea324853455672c6c (DEFAULT_TABLE table=test-table [id=aec8b423bf774e35beabf2b480fcfda8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:02.729049 26428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51dc0b437c8c49aea324853455672c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:02.734992 26486 tablet_bootstrap.cc:492] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35: Bootstrap starting.
I20260321 08:57:02.824958 26487 tablet_bootstrap.cc:492] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b: Bootstrap starting.
I20260321 08:57:02.833077 26486 tablet_bootstrap.cc:654] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:02.842178 26489 tablet_bootstrap.cc:492] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be: Bootstrap starting.
I20260321 08:57:02.843554 26487 tablet_bootstrap.cc:654] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:02.849567 26489 tablet_bootstrap.cc:654] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:02.850590 26486 tablet_bootstrap.cc:492] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35: No bootstrap required, opened a new log
I20260321 08:57:02.851097 26486 ts_tablet_manager.cc:1403] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35: Time spent bootstrapping tablet: real 0.116s user 0.007s sys 0.004s
I20260321 08:57:02.852834 26487 tablet_bootstrap.cc:492] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b: No bootstrap required, opened a new log
I20260321 08:57:02.853219 26487 ts_tablet_manager.cc:1403] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.003s
I20260321 08:57:02.853914 26486 raft_consensus.cc:359] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.854746 26486 raft_consensus.cc:385] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:02.855052 26486 raft_consensus.cc:740] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d119b009c5744457b0555896e263fa35, State: Initialized, Role: FOLLOWER
I20260321 08:57:02.855204 26487 raft_consensus.cc:359] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.855860 26487 raft_consensus.cc:385] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:02.856266 26487 raft_consensus.cc:740] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Initialized, Role: FOLLOWER
I20260321 08:57:02.855779 26486 consensus_queue.cc:260] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.856863 26487 consensus_queue.cc:260] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.860078 26486 ts_tablet_manager.cc:1434] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260321 08:57:02.866869 26487 ts_tablet_manager.cc:1434] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260321 08:57:02.873095 26486 tablet_bootstrap.cc:492] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35: Bootstrap starting.
I20260321 08:57:02.874241 26487 tablet_bootstrap.cc:492] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b: Bootstrap starting.
I20260321 08:57:02.879604 26486 tablet_bootstrap.cc:654] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:02.882573 26487 tablet_bootstrap.cc:654] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:02.886440 26489 tablet_bootstrap.cc:492] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be: No bootstrap required, opened a new log
I20260321 08:57:02.886977 26489 ts_tablet_manager.cc:1403] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.012s
I20260321 08:57:02.889992 26489 raft_consensus.cc:359] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.891682 26489 raft_consensus.cc:385] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:02.892041 26489 raft_consensus.cc:740] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Initialized, Role: FOLLOWER
I20260321 08:57:02.892835 26489 consensus_queue.cc:260] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.894735 26493 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:02.895301 26493 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.896723 26486 tablet_bootstrap.cc:492] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35: No bootstrap required, opened a new log
I20260321 08:57:02.897296 26486 ts_tablet_manager.cc:1403] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.004s
I20260321 08:57:02.898988 26487 tablet_bootstrap.cc:492] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b: No bootstrap required, opened a new log
I20260321 08:57:02.899477 26487 ts_tablet_manager.cc:1403] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.001s
I20260321 08:57:02.900981 26486 raft_consensus.cc:359] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.901810 26486 raft_consensus.cc:385] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:02.902539 26487 raft_consensus.cc:359] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.902927 26486 raft_consensus.cc:740] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d119b009c5744457b0555896e263fa35, State: Initialized, Role: FOLLOWER
I20260321 08:57:02.903244 26487 raft_consensus.cc:385] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:02.903597 26487 raft_consensus.cc:740] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Initialized, Role: FOLLOWER
I20260321 08:57:02.904040 26486 consensus_queue.cc:260] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.904264 26487 consensus_queue.cc:260] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.908527 26493 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:02.913973 26487 ts_tablet_manager.cc:1434] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b: Time spent starting tablet: real 0.014s user 0.010s sys 0.001s
I20260321 08:57:02.919271 26486 ts_tablet_manager.cc:1434] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260321 08:57:02.927994 26487 tablet_bootstrap.cc:492] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b: Bootstrap starting.
I20260321 08:57:02.930680 26492 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:02.931257 26492 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.933725 26492 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:02.933784 26489 ts_tablet_manager.cc:1434] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be: Time spent starting tablet: real 0.046s user 0.011s sys 0.000s
I20260321 08:57:02.936815 26489 tablet_bootstrap.cc:492] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be: Bootstrap starting.
I20260321 08:57:02.937080 26487 tablet_bootstrap.cc:654] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:02.947553 26489 tablet_bootstrap.cc:654] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:02.954741 26486 tablet_bootstrap.cc:492] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35: Bootstrap starting.
I20260321 08:57:02.956765 26492 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:02.957921 26492 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:02.959925 26492 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:02.997418 26493 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:02.998032 26493 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.010939 26496 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:03.011566 26496 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.024956 26493 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:03.022672 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:03.028669 26496 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:03.029026 26440 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 0.
I20260321 08:57:03.043143 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:03.052891 26487 tablet_bootstrap.cc:492] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b: No bootstrap required, opened a new log
I20260321 08:57:03.053931 26489 tablet_bootstrap.cc:492] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be: No bootstrap required, opened a new log
I20260321 08:57:03.056054 26489 ts_tablet_manager.cc:1403] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.016s
I20260321 08:57:03.056044 26487 ts_tablet_manager.cc:1403] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b: Time spent bootstrapping tablet: real 0.128s user 0.030s sys 0.035s
I20260321 08:57:03.059842 26486 tablet_bootstrap.cc:654] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:03.056973 26241 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
W20260321 08:57:03.064384 26241 leader_election.cc:343] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993): Illegal state: must be running to vote when last-logged opid is not known
I20260321 08:57:03.065132 26493 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:57:03.066074 26493 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:03.066540 26493 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:03.068017 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:03.069459 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:03.070361 26362 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 0.
I20260321 08:57:03.071103 26361 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 0.
I20260321 08:57:03.068027 26487 raft_consensus.cc:359] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.072172 26487 raft_consensus.cc:385] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:03.072481 26487 raft_consensus.cc:740] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Initialized, Role: FOLLOWER
I20260321 08:57:03.073132 26487 consensus_queue.cc:260] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.076552 26244 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters: dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:03.079547 26200 catalog_manager.cc:1269] Loaded cluster ID: 9774d2365c48473a92ff616a29d5cfc9
I20260321 08:57:03.079856 26200 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: loading cluster ID for follower catalog manager: success
I20260321 08:57:03.083513 26499 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:57:03.083966 26499 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:03.084288 26499 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:03.086850 26489 raft_consensus.cc:359] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.088959 26489 raft_consensus.cc:385] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:03.089283 26489 raft_consensus.cc:740] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Initialized, Role: FOLLOWER
I20260321 08:57:03.089259 26487 ts_tablet_manager.cc:1434] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20260321 08:57:03.091051 26489 consensus_queue.cc:260] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.086040 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:03.092904 26440 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 0.
I20260321 08:57:03.096757 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:03.097355 26440 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 0.
I20260321 08:57:03.098910 26325 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters:
I20260321 08:57:03.099756 26486 tablet_bootstrap.cc:492] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35: No bootstrap required, opened a new log
I20260321 08:57:03.099980 26325 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters:
I20260321 08:57:03.100277 26486 ts_tablet_manager.cc:1403] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35: Time spent bootstrapping tablet: real 0.146s user 0.004s sys 0.012s
I20260321 08:57:03.100572 26492 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:57:03.101181 26492 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:03.101512 26492 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:03.109820 26489 ts_tablet_manager.cc:1434] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be: Time spent starting tablet: real 0.053s user 0.006s sys 0.002s
I20260321 08:57:03.112833 26489 tablet_bootstrap.cc:492] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be: Bootstrap starting.
I20260321 08:57:03.114274 26225 catalog_manager.cc:1269] Loaded cluster ID: 9774d2365c48473a92ff616a29d5cfc9
I20260321 08:57:03.114578 26225 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74: loading cluster ID for follower catalog manager: success
I20260321 08:57:03.117268 26486 raft_consensus.cc:359] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.117998 26486 raft_consensus.cc:385] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:03.122287 26200 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: acquiring CA information for follower catalog manager: success
I20260321 08:57:03.108153 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:03.120127 26486 raft_consensus.cc:740] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d119b009c5744457b0555896e263fa35, State: Initialized, Role: FOLLOWER
I20260321 08:57:03.126945 26486 consensus_queue.cc:260] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.138797 26200 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260321 08:57:03.147364 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:03.147938 26499 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.150671 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:03.157390 26492 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.158668 26361 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd3e8a46765d4e07b3d1087aca0be2be in current term 1: Already voted for candidate d119b009c5744457b0555896e263fa35 in this term.
I20260321 08:57:03.159855 26489 tablet_bootstrap.cc:654] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:03.162205 26493 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.163307 26284 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd3e8a46765d4e07b3d1087aca0be2be in current term 1: Already voted for candidate de2947147ecb4d6eafa4aa039126f25b in this term.
I20260321 08:57:03.164964 26404 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be; no voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:03.166949 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:03.167543 26493 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 election: Requested vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:03.167685 26284 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 1: Already voted for candidate de2947147ecb4d6eafa4aa039126f25b in this term.
I20260321 08:57:03.168498 26493 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:03.168553 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:03.167907 26496 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:03.173650 26499 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 election: Requested vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:03.172607 26493 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.172788 26509 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:57:03.175916 26492 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 1 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:03.178952 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:03.183323 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:03.183508 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:03.183774 26486 ts_tablet_manager.cc:1434] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35: Time spent starting tablet: real 0.083s user 0.014s sys 0.004s
I20260321 08:57:03.184021 26439 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:03.184283 26361 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate de2947147ecb4d6eafa4aa039126f25b in current term 1: Already voted for candidate d119b009c5744457b0555896e263fa35 in this term.
I20260321 08:57:03.180539 26362 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:03.186229 26438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:03.189957 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:03.190788 26283 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 1: Already voted for candidate de2947147ecb4d6eafa4aa039126f25b in this term.
I20260321 08:57:03.191771 26489 tablet_bootstrap.cc:492] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be: No bootstrap required, opened a new log
I20260321 08:57:03.192035 26284 raft_consensus.cc:2393] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 1: Already voted for candidate de2947147ecb4d6eafa4aa039126f25b in this term.
I20260321 08:57:03.192243 26489 ts_tablet_manager.cc:1403] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be: Time spent bootstrapping tablet: real 0.080s user 0.015s sys 0.008s
I20260321 08:57:03.196478 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:03.197243 26284 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 1: Already voted for candidate de2947147ecb4d6eafa4aa039126f25b in this term.
I20260321 08:57:03.198226 26496 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 08:57:03.199093 26439 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 1.
I20260321 08:57:03.199258 26225 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74: acquiring CA information for follower catalog manager: success
I20260321 08:57:03.199764 26362 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 1.
I20260321 08:57:03.200335 26509 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:03.200755 26509 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:03.201571 26489 raft_consensus.cc:359] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.202323 26489 raft_consensus.cc:385] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:03.202605 26489 raft_consensus.cc:740] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Initialized, Role: FOLLOWER
I20260321 08:57:03.203254 26489 consensus_queue.cc:260] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.204748 26440 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 1.
I20260321 08:57:03.205525 26489 ts_tablet_manager.cc:1434] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260321 08:57:03.206513 26241 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:03.207134 26499 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:03.207536 26499 raft_consensus.cc:697] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 1 LEADER]: Becoming Leader. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:03.207804 26241 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35
I20260321 08:57:03.208320 26499 consensus_queue.cc:237] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.220486 26492 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:03.221078 26492 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.222013 26511 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:03.222483 26511 raft_consensus.cc:697] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 LEADER]: Becoming Leader. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:03.223022 26492 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:03.224865 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:03.225638 26440 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 0.
I20260321 08:57:03.226449 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:03.226699 26325 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters:
I20260321 08:57:03.227339 26362 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 0.
I20260321 08:57:03.227456 26493 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:03.227970 26492 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:57:03.228401 26492 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:03.228737 26492 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:03.230707 26225 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260321 08:57:03.232677 26509 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.223234 26511 consensus_queue.cc:237] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.226629 26326 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:03.236505 26510 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260321 08:57:03.237568 26509 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 2 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:03.238186 26244 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:03.235637 26492 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.239296 26511 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:57:03.239706 26511 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:03.240051 26511 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:03.241456 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:03.242110 26440 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 0.
I20260321 08:57:03.244933 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:03.245479 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:03.246089 26439 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:03.246505 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:03.246357 26511 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.247550 26284 raft_consensus.cc:2393] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 1: Already voted for candidate de2947147ecb4d6eafa4aa039126f25b in this term.
I20260321 08:57:03.248183 26511 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 election: Requested vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:03.248945 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:03.249439 26440 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:03.253050 26438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:03.253230 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:03.254074 26284 raft_consensus.cc:2393] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 1: Already voted for candidate de2947147ecb4d6eafa4aa039126f25b in this term.
I20260321 08:57:03.254302 26440 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 2.
I20260321 08:57:03.255160 26492 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 1 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:03.255432 26325 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters:
I20260321 08:57:03.260321 26492 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 08:57:03.261122 26492 raft_consensus.cc:697] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 2 LEADER]: Becoming Leader. State: Replica: d119b009c5744457b0555896e263fa35, State: Running, Role: LEADER
I20260321 08:57:03.261904 26492 consensus_queue.cc:237] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.262197 26146 catalog_manager.cc:5671] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b reported cstate change: term changed from 0 to 1, leader changed from <none> to de2947147ecb4d6eafa4aa039126f25b (127.24.0.129). New cstate: current_term: 1 leader_uuid: "de2947147ecb4d6eafa4aa039126f25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: UNKNOWN } } }
I20260321 08:57:03.256161 26439 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 1.
I20260321 08:57:03.255827 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:03.268651 26362 raft_consensus.cc:2393] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate de2947147ecb4d6eafa4aa039126f25b in current term 1: Already voted for candidate d119b009c5744457b0555896e263fa35 in this term.
I20260321 08:57:03.268893 26325 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters: de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:03.269608 26244 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:03.269840 26509 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:03.270414 26509 raft_consensus.cc:697] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 1 LEADER]: Becoming Leader. State: Replica: d119b009c5744457b0555896e263fa35, State: Running, Role: LEADER
I20260321 08:57:03.271270 26509 consensus_queue.cc:237] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.272639 26499 raft_consensus.cc:2749] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260321 08:57:03.290975 26147 catalog_manager.cc:5671] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 reported cstate change: term changed from 1 to 2, leader changed from de2947147ecb4d6eafa4aa039126f25b (127.24.0.129) to d119b009c5744457b0555896e263fa35 (127.24.0.130). New cstate: current_term: 2 leader_uuid: "d119b009c5744457b0555896e263fa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: UNKNOWN } } }
I20260321 08:57:03.297009 26146 catalog_manager.cc:5671] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b reported cstate change: term changed from 0 to 1, leader changed from <none> to de2947147ecb4d6eafa4aa039126f25b (127.24.0.129). New cstate: current_term: 1 leader_uuid: "de2947147ecb4d6eafa4aa039126f25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: UNKNOWN } } }
I20260321 08:57:03.303710 26439 raft_consensus.cc:1240] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Rejecting Update request from peer de2947147ecb4d6eafa4aa039126f25b for earlier term 1. Current term is 2. Ops: []
I20260321 08:57:03.305215 26499 consensus_queue.cc:1059] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:03.306087 26514 raft_consensus.cc:3055] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 1 LEADER]: Stepping down as leader of term 1
I20260321 08:57:03.306470 26514 raft_consensus.cc:740] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:03.307235 26514 consensus_queue.cc:260] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.308732 26514 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:03.311996 26147 catalog_manager.cc:5671] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 reported cstate change: term changed from 0 to 1, leader changed from <none> to d119b009c5744457b0555896e263fa35 (127.24.0.130). New cstate: current_term: 1 leader_uuid: "d119b009c5744457b0555896e263fa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: UNKNOWN } } }
I20260321 08:57:03.318133 26499 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:03.339854 26514 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:03.360081 26284 raft_consensus.cc:1275] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Refusing update from remote peer d119b009c5744457b0555896e263fa35: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260321 08:57:03.360512 26284 pending_rounds.cc:85] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b: Aborting all ops after (but not including) 0
I20260321 08:57:03.360790 26284 pending_rounds.cc:107] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260321 08:57:03.361066 26284 raft_consensus.cc:2889] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 08:57:03.362754 26492 consensus_queue.cc:1048] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:03.375517 26439 raft_consensus.cc:1275] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Refusing update from remote peer d119b009c5744457b0555896e263fa35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260321 08:57:03.377743 26509 consensus_queue.cc:1048] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:03.394176 26510 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:03.402249 26492 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:03.658576 26283 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb"
dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
from {username='slave'} at 127.0.0.1:42850
I20260321 08:57:03.659145 26283 raft_consensus.cc:591] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 LEADER]: Received request to step down
I20260321 08:57:03.659390 26283 raft_consensus.cc:3055] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 LEADER]: Stepping down as leader of term 1
I20260321 08:57:03.659605 26283 raft_consensus.cc:740] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:03.660189 26283 consensus_queue.cc:260] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.661033 26283 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:03.801867 26496 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:03.802357 26496 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.803846 26496 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:03.804651 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:03.804850 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:03.805153 26283 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 2.
I20260321 08:57:03.806016 26404 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:03.806614 26496 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 08:57:03.806877 26496 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Starting leader election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:03.807147 26496 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:03.811952 26496 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.813479 26496 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 2 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:03.814209 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:03.814288 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:03.819693 26283 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 2.
I20260321 08:57:03.820600 26404 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35
I20260321 08:57:03.821376 26496 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Leader election won for term 2
I20260321 08:57:03.822290 26496 raft_consensus.cc:697] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 2 LEADER]: Becoming Leader. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:03.823122 26496 consensus_queue.cc:237] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.830833 26145 catalog_manager.cc:5671] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be reported cstate change: term changed from 1 to 2, leader changed from de2947147ecb4d6eafa4aa039126f25b (127.24.0.129) to dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131). New cstate: current_term: 2 leader_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: HEALTHY } } }
I20260321 08:57:03.840206 26540 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:03.840806 26540 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.842865 26540 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:03.843416 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:03.844483 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:03.846374 26283 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 2: Already voted for candidate dd3e8a46765d4e07b3d1087aca0be2be in this term.
I20260321 08:57:03.847410 26326 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:03.848850 26540 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:03.855214 26540 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 08:57:03.881629 26362 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a"
dest_uuid: "d119b009c5744457b0555896e263fa35"
from {username='slave'} at 127.0.0.1:41964
I20260321 08:57:03.882117 26362 raft_consensus.cc:591] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 1 LEADER]: Received request to step down
I20260321 08:57:03.882354 26362 raft_consensus.cc:3055] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 08:57:03.882586 26362 raft_consensus.cc:740] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d119b009c5744457b0555896e263fa35, State: Running, Role: LEADER
I20260321 08:57:03.883229 26362 consensus_queue.cc:260] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:03.884148 26362 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:03.921286 26283 raft_consensus.cc:1275] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Refusing update from remote peer dd3e8a46765d4e07b3d1087aca0be2be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260321 08:57:03.922442 26496 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 08:57:03.929281 26362 raft_consensus.cc:1275] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Refusing update from remote peer dd3e8a46765d4e07b3d1087aca0be2be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260321 08:57:03.933151 26553 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 08:57:04.027580 26499 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:04.028081 26499 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:04.029641 26499 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:04.030463 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:04.030757 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:04.031107 26362 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 2.
I20260321 08:57:04.032140 26244 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters: dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:04.032963 26499 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 08:57:04.033291 26499 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Starting leader election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:04.033537 26499 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:04.038604 26499 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:04.040330 26499 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 2 election: Requested vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:04.041347 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:04.041703 26438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:04.059106 26362 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 2.
I20260321 08:57:04.060954 26244 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters: dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:04.062726 26499 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Leader election won for term 2
I20260321 08:57:04.068230 26499 raft_consensus.cc:697] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 2 LEADER]: Becoming Leader. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:04.069850 26499 consensus_queue.cc:237] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:04.087551 26561 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:04.088322 26561 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:04.091578 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:04.099263 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:04.100087 26362 raft_consensus.cc:2393] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd3e8a46765d4e07b3d1087aca0be2be in current term 2: Already voted for candidate de2947147ecb4d6eafa4aa039126f25b in this term.
I20260321 08:57:04.101728 26405 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be; no voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:04.102538 26560 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:04.103323 26561 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:04.108938 26560 raft_consensus.cc:2749] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 08:57:04.183733 26438 raft_consensus.cc:1275] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Refusing update from remote peer de2947147ecb4d6eafa4aa039126f25b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260321 08:57:04.185067 26562 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 08:57:04.186033 26362 raft_consensus.cc:1275] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Refusing update from remote peer de2947147ecb4d6eafa4aa039126f25b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260321 08:57:04.189814 26562 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 08:57:04.199533 26362 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c"
dest_uuid: "d119b009c5744457b0555896e263fa35"
from {username='slave'} at 127.0.0.1:41964
I20260321 08:57:04.200134 26362 raft_consensus.cc:591] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 2 LEADER]: Received request to step down
I20260321 08:57:04.200460 26362 raft_consensus.cc:3055] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 08:57:04.200762 26362 raft_consensus.cc:740] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d119b009c5744457b0555896e263fa35, State: Running, Role: LEADER
I20260321 08:57:04.201459 26362 consensus_queue.cc:260] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:04.202623 26362 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:04.206804 26147 catalog_manager.cc:5671] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be reported cstate change: term changed from 1 to 2, leader changed from d119b009c5744457b0555896e263fa35 (127.24.0.130) to de2947147ecb4d6eafa4aa039126f25b (127.24.0.129). New cstate: current_term: 2 leader_uuid: "de2947147ecb4d6eafa4aa039126f25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } } }
I20260321 08:57:04.210186 26564 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9af081622eaa4661a29feb3bf5d18cc8)
I20260321 08:57:04.210750 26564 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:04.213284 26027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
I20260321 08:57:04.220798 26157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:04.223572 26067 leader_election.cc:304] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9e21016d934beab3beb647aa48006b; no voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8
I20260321 08:57:04.224205 26564 leader_election.cc:290] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:04.224727 26564 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 08:57:04.288602 26540 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:04.289186 26540 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:04.291203 26540 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:04.304889 26438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:04.310927 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:04.312427 26326 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:04.313556 26540 raft_consensus.cc:2749] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 08:57:04.337499 26560 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:04.337952 26560 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:04.339645 26560 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:04.725455 26564 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9af081622eaa4661a29feb3bf5d18cc8)
I20260321 08:57:04.726073 26564 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:04.729214 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
I20260321 08:57:04.729892 26026 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9e21016d934beab3beb647aa48006b in term 1.
I20260321 08:57:04.731353 26066 leader_election.cc:304] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 5c9e21016d934beab3beb647aa48006b; no voters:
I20260321 08:57:04.732475 26564 leader_election.cc:290] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:04.733023 26564 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 08:57:04.733352 26564 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9af081622eaa4661a29feb3bf5d18cc8)
I20260321 08:57:04.734736 26564 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:04.822499 26564 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:04.910035 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74"
I20260321 08:57:04.910727 26026 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:04.917632 26026 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9e21016d934beab3beb647aa48006b in term 2.
I20260321 08:57:04.918854 26066 leader_election.cc:304] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 5c9e21016d934beab3beb647aa48006b; no voters:
I20260321 08:57:04.919965 26565 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 2 FOLLOWER]: Leader election won for term 2
I20260321 08:57:04.931504 26156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8"
I20260321 08:57:04.933362 26564 leader_election.cc:290] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 2 election: Requested vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
W20260321 08:57:04.967018 26405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.130:41799, user_credentials={real_user=slave}} blocked reactor thread for 72989.9us
I20260321 08:57:04.986905 26565 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 2 LEADER]: Becoming Leader. State: Replica: 5c9e21016d934beab3beb647aa48006b, State: Running, Role: LEADER
I20260321 08:57:04.988085 26565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:05.041071 26156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
W20260321 08:57:05.064338 26362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41944 (request call id 10) took 724 ms (client timeout 294 ms). Trace:
W20260321 08:57:05.064877 26362 rpcz_store.cc:269] 0321 08:57:04.340289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:05.063959 (+723670us) service_pool.cc:212] Skipping call since client already timed out
0321 08:57:05.064258 (+ 299us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260321 08:57:05.088824 26156 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 08:57:05.089277 26156 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9af081622eaa4661a29feb3bf5d18cc8, State: Running, Role: LEADER
I20260321 08:57:05.090104 26156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:05.091320 26156 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 1 FOLLOWER]: Advancing to term 2
W20260321 08:57:05.098629 26132 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 08:57:05.101341 26156 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 2 FOLLOWER]: Refusing update from remote peer 5c9e21016d934beab3beb647aa48006b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20260321 08:57:05.102072 26566 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:05.102548 26566 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:05.103163 26565 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260321 08:57:05.105446 26092 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b" is_pre_election: true
I20260321 08:57:05.105758 26026 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Refusing update from remote peer 5c9e21016d934beab3beb647aa48006b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20260321 08:57:05.107730 26565 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260321 08:57:05.108906 26156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:05.108229 26566 leader_election.cc:290] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:05.110514 26002 leader_election.cc:304] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74; no voters: 5c9e21016d934beab3beb647aa48006b, 9af081622eaa4661a29feb3bf5d18cc8
I20260321 08:57:05.112489 26566 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 08:57:05.112356 26570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9e21016d934beab3beb647aa48006b. Latest consensus state: current_term: 2 leader_uuid: "5c9e21016d934beab3beb647aa48006b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:05.113351 26570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:05.120772 26540 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:05.121304 26540 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.123899 26540 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:05.124524 26438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:05.129011 26571 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:05.129515 26571 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.131131 26571 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:05.131542 26571 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:05.131873 26571 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.133553 26571 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
W20260321 08:57:05.142343 26157 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47108 (request call id 2) took 408 ms (client timeout 277 ms). Trace:
W20260321 08:57:05.142735 26157 rpcz_store.cc:269] 0321 08:57:04.734508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:05.142164 (+407656us) service_pool.cc:212] Skipping call since client already timed out
0321 08:57:05.142282 (+ 118us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260321 08:57:05.149899 26564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9e21016d934beab3beb647aa48006b. Latest consensus state: current_term: 2 leader_uuid: "5c9e21016d934beab3beb647aa48006b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:05.150647 26564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:05.155498 26553 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:05.156071 26553 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.158592 26553 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:05.170390 26092 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 2 LEADER]: Rejecting Update request from peer 9af081622eaa4661a29feb3bf5d18cc8 for earlier term 1. Current term is 2. Ops: []
I20260321 08:57:05.176487 26254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9e21016d934beab3beb647aa48006b. Latest consensus state: current_term: 2 leader_uuid: "5c9e21016d934beab3beb647aa48006b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:05.177269 26254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: FOLLOWER
W20260321 08:57:05.179028 26001 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.006s sys 0.003s
W20260321 08:57:05.179373 26001 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.758s user 0.006s sys 0.003s
I20260321 08:57:05.182317 26213 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260321 08:57:05.194226 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:05.194931 26362 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 3.
I20260321 08:57:05.195960 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:05.196557 26361 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 2.
I20260321 08:57:05.197984 26405 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters:
I20260321 08:57:05.198894 26405 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters:
I20260321 08:57:05.200071 26553 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 08:57:05.200417 26553 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Starting leader election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:05.200819 26553 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:05.203011 26438 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb"
dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
from {username='slave'} at 127.0.0.1:33172
I20260321 08:57:05.203598 26438 raft_consensus.cc:591] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 2 LEADER]: Received request to step down
I20260321 08:57:05.203953 26438 raft_consensus.cc:3055] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 2 LEADER]: Stepping down as leader of term 2
I20260321 08:57:05.204232 26438 raft_consensus.cc:740] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:05.204924 26438 consensus_queue.cc:260] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.209992 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:05.210536 26283 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 2.
I20260321 08:57:05.207821 26553 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.216526 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:05.217057 26283 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:05.218009 26553 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 3 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:05.218425 26553 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 08:57:05.218643 26553 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Starting leader election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:05.218924 26553 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:05.223814 26360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:05.225052 26281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:05.225803 26281 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 2.
I20260321 08:57:05.226166 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:05.226747 26361 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 3.
I20260321 08:57:05.230116 26553 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.230595 26326 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters: dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:05.231415 26540 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 08:57:05.231771 26540 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Starting leader election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:05.231864 26244 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:05.232205 26540 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:05.232903 26281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:05.234212 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:05.236493 26360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:05.238070 26405 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be; no voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:05.238986 26553 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 3 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:05.239774 26553 raft_consensus.cc:2749] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 08:57:05.239619 26540 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.241252 26283 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 3.
I20260321 08:57:05.241726 26499 raft_consensus.cc:2764] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260321 08:57:05.242066 26540 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:05.242286 26281 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:05.242887 26281 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:05.247740 26404 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:05.248956 26561 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Leader election won for term 3
I20260321 08:57:05.249235 26281 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 3.
W20260321 08:57:05.250315 26406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 08:57:05.250835 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:05.251802 26326 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:05.252272 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:05.252883 26540 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 08:57:05.261531 26561 raft_consensus.cc:697] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 3 LEADER]: Becoming Leader. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:05.262414 26561 consensus_queue.cc:237] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.266736 26558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "5c9e21016d934beab3beb647aa48006b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:05.267536 26558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:05.267697 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:05.268358 26284 raft_consensus.cc:2376] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 3. Re-sending same reply.
I20260321 08:57:05.268776 26540 raft_consensus.cc:697] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 3 LEADER]: Becoming Leader. State: Replica: d119b009c5744457b0555896e263fa35, State: Running, Role: LEADER
I20260321 08:57:05.288957 26573 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:05.277160 26147 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47154 (request call id 29) took 1017 ms. Trace:
I20260321 08:57:05.294624 26570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "5c9e21016d934beab3beb647aa48006b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:05.294235 26147 rpcz_store.cc:276] 0321 08:57:04.259531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:04.259762 (+ 231us) service_pool.cc:225] Handling call
0321 08:57:05.277081 (+1017319us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1014218,"reported_tablets":1}
I20260321 08:57:05.295455 26570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:05.297482 26438 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:05.284139 26145 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47132 (request call id 28) took 1048 ms. Trace:
I20260321 08:57:05.300372 26145 rpcz_store.cc:276] 0321 08:57:04.225738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:04.225953 (+ 215us) service_pool.cc:225] Handling call
0321 08:57:05.273673 (+1047720us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2198,"reported_tablets":1}
I20260321 08:57:05.287585 26361 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 3.
I20260321 08:57:05.308400 26565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "5c9e21016d934beab3beb647aa48006b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:05.309233 26565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:05.310750 26573 catalog_manager.cc:679] Loaded metadata for table test-table [id=aec8b423bf774e35beabf2b480fcfda8]
I20260321 08:57:05.316056 26565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "5c9e21016d934beab3beb647aa48006b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:05.316771 26565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:05.322365 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:05.324059 26437 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:05.326205 26241 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:05.327708 26439 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 3.
I20260321 08:57:05.323671 26540 consensus_queue.cc:237] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.329634 26438 raft_consensus.cc:1240] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Rejecting Update request from peer de2947147ecb4d6eafa4aa039126f25b for earlier term 2. Current term is 3. Ops: []
I20260321 08:57:05.328045 26499 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 08:57:05.331192 26562 consensus_queue.cc:1059] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 08:57:05.343638 26562 raft_consensus.cc:3055] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 2 LEADER]: Stepping down as leader of term 2
I20260321 08:57:05.344084 26562 raft_consensus.cc:740] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:05.346344 26562 consensus_queue.cc:260] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.347404 26562 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:05.348230 26361 raft_consensus.cc:1275] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Refusing update from remote peer dd3e8a46765d4e07b3d1087aca0be2be: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260321 08:57:05.350099 26113 debug-util.cc:398] Leaking SignalData structure 0x7b080015d660 after lost signal to thread 24581
W20260321 08:57:05.350977 26113 debug-util.cc:398] Leaking SignalData structure 0x7b080015cec0 after lost signal to thread 26051
W20260321 08:57:05.351683 26113 debug-util.cc:398] Leaking SignalData structure 0x7b080009d660 after lost signal to thread 26116
W20260321 08:57:05.352404 26113 debug-util.cc:398] Leaking SignalData structure 0x7b0800245020 after lost signal to thread 26181
W20260321 08:57:05.353464 26113 debug-util.cc:398] Leaking SignalData structure 0x7b08001bae60 after lost signal to thread 26308
W20260321 08:57:05.354475 26113 debug-util.cc:398] Leaking SignalData structure 0x7b08000d9340 after lost signal to thread 26386
W20260321 08:57:05.356031 26244 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 08:57:05.360879 26575 consensus_queue.cc:1048] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 08:57:05.364996 26284 raft_consensus.cc:1275] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Refusing update from remote peer dd3e8a46765d4e07b3d1087aca0be2be: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 08:57:05.365561 26573 tablet_loader.cc:96] loaded metadata for tablet 3effe2075d034127a1ac31b507b8119a (table test-table [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:05.367008 26573 tablet_loader.cc:96] loaded metadata for tablet 51dc0b437c8c49aea324853455672c6c (table test-table [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:05.367566 26561 consensus_queue.cc:1048] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 08:57:05.383137 26573 tablet_loader.cc:96] loaded metadata for tablet b4eff2c92f994bb2a318dfe6097efacb (table test-table [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:05.384764 26573 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:05.390542 26573 catalog_manager.cc:1269] Loaded cluster ID: 9774d2365c48473a92ff616a29d5cfc9
I20260321 08:57:05.390818 26573 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:05.399526 26573 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:05.403803 26573 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: Loaded TSK: 0
I20260321 08:57:05.418716 26573 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:05.434587 26540 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:05.434885 26283 raft_consensus.cc:1275] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Refusing update from remote peer d119b009c5744457b0555896e263fa35: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260321 08:57:05.435149 26540 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.437975 26540 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:05.438203 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:05.439072 26439 raft_consensus.cc:2393] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 3: Already voted for candidate dd3e8a46765d4e07b3d1087aca0be2be in this term.
I20260321 08:57:05.443089 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:05.443795 26283 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 3.
I20260321 08:57:05.445251 26326 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters: dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:05.445839 26580 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260321 08:57:05.457710 26580 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 08:57:05.458097 26580 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Starting leader election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:05.457374 26082 catalog_manager.cc:5671] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b reported cstate change: term changed from 2 to 3, leader changed from d119b009c5744457b0555896e263fa35 (127.24.0.130) to dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131). New cstate: current_term: 3 leader_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } } }
I20260321 08:57:05.458437 26580 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:05.461015 26466 heartbeater.cc:499] Master 127.24.0.189:38105 was elected leader, sending a full tablet report...
I20260321 08:57:05.464627 26439 raft_consensus.cc:1275] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Refusing update from remote peer d119b009c5744457b0555896e263fa35: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260321 08:57:05.466049 26540 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 08:57:05.469491 26575 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:05.472894 26575 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.496369 26575 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:05.501276 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:05.501998 26283 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 3.
I20260321 08:57:05.516086 26404 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:05.523974 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:05.521035 26575 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 08:57:05.521265 26580 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.525502 26575 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:05.528164 26575 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:57:05.540879 26361 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 3.
I20260321 08:57:05.547434 26580 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:05.548563 26438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:05.550246 26080 catalog_manager.cc:5671] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 reported cstate change: term changed from 2 to 3, leader changed from dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131) to d119b009c5744457b0555896e263fa35 (127.24.0.130). New cstate: current_term: 3 leader_uuid: "d119b009c5744457b0555896e263fa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: HEALTHY } } }
I20260321 08:57:05.553198 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:05.554051 26575 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.555655 26438 raft_consensus.cc:2368] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 for earlier term 3. Current term is 4.
I20260321 08:57:05.557550 26575 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 4 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:05.557658 26325 leader_election.cc:400] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 election: Vote denied by peer dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993) with higher term. Message: Invalid argument: T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 for earlier term 3. Current term is 4.
I20260321 08:57:05.558348 26325 leader_election.cc:403] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260321 08:57:05.558710 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:05.558961 26580 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:57:05.565820 26580 raft_consensus.cc:2749] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993) with higher term. Message: Invalid argument: T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 for earlier term 3. Current term is 4.
I20260321 08:57:05.565784 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:05.572124 26283 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 3.
I20260321 08:57:05.581537 26361 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 4.
I20260321 08:57:05.582628 26405 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters: de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:05.584319 26575 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Leader election won for term 4
I20260321 08:57:05.584846 26575 raft_consensus.cc:697] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 4 LEADER]: Becoming Leader. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:05.585532 26575 consensus_queue.cc:237] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:05.621027 26310 heartbeater.cc:499] Master 127.24.0.189:38105 was elected leader, sending a full tablet report...
I20260321 08:57:05.624478 26388 heartbeater.cc:499] Master 127.24.0.189:38105 was elected leader, sending a full tablet report...
I20260321 08:57:05.636045 26078 catalog_manager.cc:5671] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be reported cstate change: term changed from 2 to 4, leader changed from de2947147ecb4d6eafa4aa039126f25b (127.24.0.129) to dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131). New cstate: current_term: 4 leader_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: HEALTHY } } }
I20260321 08:57:05.688095 26082 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49372:
table { table_id: "aec8b423bf774e35beabf2b480fcfda8" table_name: "test-table" } new_table_name: "NewTableName"
I20260321 08:57:05.698129 26283 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:57:05.715255 26361 raft_consensus.cc:1275] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Refusing update from remote peer dd3e8a46765d4e07b3d1087aca0be2be: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 08:57:05.718585 26553 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 08:57:05.754729 26082 catalog_manager.cc:5763] TS dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993) does not have the latest schema for tablet 3effe2075d034127a1ac31b507b8119a (table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8]). Expected version 1 got 0
I20260321 08:57:05.751732 24578 master.cc:562] Master@127.24.0.189:38105 shutting down...
I20260321 08:57:05.783957 26283 raft_consensus.cc:1275] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Refusing update from remote peer dd3e8a46765d4e07b3d1087aca0be2be: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 08:57:05.787189 26553 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 08:57:05.875748 26607 tablet.cc:1723] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260321 08:57:05.888167 26607 tablet.cc:1723] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260321 08:57:05.910238 26608 tablet.cc:1723] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260321 08:57:05.912501 26609 tablet.cc:1723] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260321 08:57:05.914590 26606 tablet.cc:1723] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260321 08:57:05.917415 26607 tablet.cc:1723] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260321 08:57:05.944726 26612 tablet.cc:1723] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260321 08:57:05.946983 26613 tablet.cc:1723] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260321 08:57:05.957460 26614 tablet.cc:1723] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260321 08:57:06.141171 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 2 LEADER]: Raft consensus shutting down.
I20260321 08:57:06.142023 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:06.142462 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: stopping tablet replica
W20260321 08:57:06.226961 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 08:57:06.264067 26575 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:06.264731 26575 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:06.267627 26575 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
W20260321 08:57:06.305186 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:06.392258 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:06.479223 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:06.569731 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:06.679786 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:06.769827 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 08:57:06.796346 26616 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5c9e21016d934beab3beb647aa48006b)
I20260321 08:57:06.796988 26616 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:06.799062 26616 leader_election.cc:290] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105)
W20260321 08:57:06.864838 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:06.967993 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:07.079033 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:07.090965 26134 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master [suppressed 11 similar messages]
I20260321 08:57:07.113705 26617 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:07.114362 26620 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:07.114334 26617 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.115092 26620 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.116679 26617 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:07.117039 26620 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
W20260321 08:57:07.122442 26134 leader_election.cc:336] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260321 08:57:07.122934 26405 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.003s sys 0.011s
W20260321 08:57:07.123188 26405 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.638s user 0.003s sys 0.011s
W20260321 08:57:07.123365 26405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.130:41799, user_credentials={real_user=slave}} blocked reactor thread for 638693us
W20260321 08:57:07.174093 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 08:57:07.185199 26620 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:07.185969 26620 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.188086 26620 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:07.214116 26620 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:07.215046 26620 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.216792 26620 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:07.232976 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:07.233755 26284 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 3.
I20260321 08:57:07.235831 26404 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:07.237216 26621 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 08:57:07.237639 26621 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Starting leader election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:07.237983 26621 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:57:07.244916 26621 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.247252 26621 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 4 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
W20260321 08:57:07.288245 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:07.341307 26405 leader_election.cc:336] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d119b009c5744457b0555896e263fa35 (127.24.0.130:41799): Timed out: RequestConsensusVote RPC to 127.24.0.130:41799 timed out after 0.217s (SENT)
W20260321 08:57:07.395314 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:07.507341 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 08:57:07.611286 26067 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 08:57:07.616773 26435 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
W20260321 08:57:07.624670 26325 leader_election.cc:336] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993): Timed out: RequestConsensusVote RPC to 127.24.0.131:36993 timed out after 0.241s (SENT)
W20260321 08:57:07.731457 26405 leader_election.cc:336] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d119b009c5744457b0555896e263fa35 (127.24.0.130:41799): Timed out: RequestConsensusVote RPC to 127.24.0.130:41799 timed out after 0.228s (SENT)
I20260321 08:57:07.760426 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:07.761863 26358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
W20260321 08:57:07.771713 26437 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33162 (request call id 43) took 387 ms (client timeout 241 ms). Trace:
W20260321 08:57:07.765055 26438 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33162 (request call id 42) took 640 ms (client timeout 241 ms). Trace:
I20260321 08:57:07.787912 26358 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
W20260321 08:57:07.792107 26438 rpcz_store.cc:269] 0321 08:57:07.124603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:07.764873 (+640270us) service_pool.cc:212] Skipping call since client already timed out
0321 08:57:07.764996 (+ 123us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260321 08:57:07.789394 26357 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:07.792546 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:07.793201 26284 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 4.
I20260321 08:57:07.790107 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:07.804597 26017 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:58302 (request call id 46) took 1095 ms. Trace:
I20260321 08:57:07.805035 26017 rpcz_store.cc:276] 0321 08:57:06.708901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:07.803258 (+1094357us) service_pool.cc:225] Handling call
0321 08:57:07.804529 (+ 1271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 08:57:07.807456 26359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
W20260321 08:57:07.772172 26437 rpcz_store.cc:269] 0321 08:57:07.384055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:07.771520 (+387465us) service_pool.cc:212] Skipping call since client already timed out
0321 08:57:07.771639 (+ 119us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260321 08:57:07.786330 26626 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:07.811124 26626 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.814776 26438 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:07.817451 26361 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 3.
W20260321 08:57:07.817755 26282 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42836 (request call id 36) took 625 ms (client timeout 217 ms). Trace:
W20260321 08:57:07.818182 26282 rpcz_store.cc:269] 0321 08:57:07.192170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:07.817582 (+625412us) service_pool.cc:212] Skipping call since client already timed out
0321 08:57:07.817700 (+ 118us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260321 08:57:07.836087 26627 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5c9e21016d934beab3beb647aa48006b)
I20260321 08:57:07.814275 26626 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:07.840024 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:07.840683 26283 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 3.
W20260321 08:57:07.827745 26436 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33160 (request call id 23) took 357 ms (client timeout 209 ms). Trace:
I20260321 08:57:07.836647 26627 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:07.844767 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
W20260321 08:57:07.842162 26436 rpcz_store.cc:269] 0321 08:57:07.470059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:07.827558 (+357499us) service_pool.cc:212] Skipping call since client already timed out
0321 08:57:07.827679 (+ 121us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260321 08:57:07.845355 26026 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9af081622eaa4661a29feb3bf5d18cc8 in term 2.
W20260321 08:57:07.847380 26440 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33160 (request call id 22) took 377 ms (client timeout 230 ms). Trace:
I20260321 08:57:07.849047 26326 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters: dd3e8a46765d4e07b3d1087aca0be2be
W20260321 08:57:07.849834 26440 rpcz_store.cc:269] 0321 08:57:07.469696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:07.847202 (+377506us) service_pool.cc:212] Skipping call since client already timed out
0321 08:57:07.847320 (+ 118us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260321 08:57:07.852140 26627 leader_election.cc:290] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:07.855284 26626 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 08:57:07.855707 26626 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Starting leader election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:07.856124 26626 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Advancing to term 5
W20260321 08:57:07.854749 26244 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.130:41799, user_credentials={real_user=slave}} blocked reactor thread for 502277us
W20260321 08:57:07.856302 26404 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.012s sys 0.003s
W20260321 08:57:07.858268 26404 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s user 0.012s sys 0.003s
W20260321 08:57:07.858740 26404 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.129:38365, user_credentials={real_user=slave}} blocked reactor thread for 622967us
I20260321 08:57:07.863152 26626 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.865304 26626 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 5 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:07.865906 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:07.866081 26359 raft_consensus.cc:1240] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Rejecting Update request from peer dd3e8a46765d4e07b3d1087aca0be2be for earlier term 4. Current term is 5. Ops: []
I20260321 08:57:07.867309 26621 consensus_queue.cc:1059] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 08:57:07.857743 26326 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters: dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:07.868307 26621 raft_consensus.cc:3055] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 4 LEADER]: Stepping down as leader of term 4
I20260321 08:57:07.868659 26621 raft_consensus.cc:740] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:07.869520 26621 consensus_queue.cc:260] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.870371 26360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:07.870646 26621 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:07.867467 26439 raft_consensus.cc:1240] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Rejecting Update request from peer d119b009c5744457b0555896e263fa35 for earlier term 3. Current term is 4. Ops: []
I20260321 08:57:07.876248 26626 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 08:57:07.876641 26626 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:07.876986 26626 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:57:07.902830 26130 leader_election.cc:304] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8; no voters: 5c9e21016d934beab3beb647aa48006b
I20260321 08:57:07.911788 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:07.912942 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:07.913476 26155 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e553cc2497492a9aa4dd5cb1d3ae74 in term 2.
I20260321 08:57:07.916093 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:07.927121 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:07.926970 26626 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.929986 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:07.922886 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:07.932289 26440 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 5.
I20260321 08:57:07.934044 26002 leader_election.cc:304] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8; no voters:
I20260321 08:57:07.935267 26241 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:07.937366 26627 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 08:57:07.937836 26627 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5c9e21016d934beab3beb647aa48006b)
I20260321 08:57:07.937973 26629 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 08:57:07.935672 26436 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de2947147ecb4d6eafa4aa039126f25b in current term 4: Already voted for candidate dd3e8a46765d4e07b3d1087aca0be2be in this term.
I20260321 08:57:07.938638 26626 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 4 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:07.938426 26629 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5c9e21016d934beab3beb647aa48006b)
I20260321 08:57:07.938310 26627 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:07.943173 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:07.945650 26629 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 2 FOLLOWER]: Advancing to term 3
I20260321 08:57:07.959540 26627 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:07.959863 26404 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be; no voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:07.966754 26621 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 08:57:07.968252 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:07.970026 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:07.971457 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:07.973867 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:07.976600 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8"
I20260321 08:57:07.977275 26627 leader_election.cc:290] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 3 election: Requested vote from peers 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
W20260321 08:57:07.978940 26004 leader_election.cc:336] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260321 08:57:07.980825 26244 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:07.985607 26629 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:07.990777 26632 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 08:57:07.991197 26632 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Starting leader election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:07.984148 26631 consensus_queue.cc:1059] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 08:57:07.992106 26155 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 42e553cc2497492a9aa4dd5cb1d3ae74 in current term 3: Already voted for candidate 9af081622eaa4661a29feb3bf5d18cc8 in this term.
I20260321 08:57:07.992751 26626 raft_consensus.cc:3055] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 08:57:07.993100 26626 raft_consensus.cc:740] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d119b009c5744457b0555896e263fa35, State: Running, Role: LEADER
I20260321 08:57:07.994000 26626 consensus_queue.cc:260] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.994947 26362 raft_consensus.cc:1240] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Rejecting Update request from peer dd3e8a46765d4e07b3d1087aca0be2be for earlier term 3. Current term is 4. Ops: []
I20260321 08:57:07.994992 26626 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 3 FOLLOWER]: Advancing to term 4
W20260321 08:57:07.991212 26241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.131:36993, user_credentials={real_user=slave}} blocked reactor thread for 56295.1us
I20260321 08:57:07.998342 26622 consensus_queue.cc:1059] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 08:57:07.999300 26622 raft_consensus.cc:3055] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 3 LEADER]: Stepping down as leader of term 3
I20260321 08:57:07.999589 26622 raft_consensus.cc:740] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:07.999866 26241 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:08.000371 26622 consensus_queue.cc:260] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:07.991559 26632 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:08.001653 26622 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:57:07.985857 26326 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:08.007540 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74"
I20260321 08:57:08.008152 26631 raft_consensus.cc:2749] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 08:57:08.008376 26026 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9af081622eaa4661a29feb3bf5d18cc8 in current term 3: Already voted for candidate 42e553cc2497492a9aa4dd5cb1d3ae74 in this term.
I20260321 08:57:08.021904 26326 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:08.022076 26633 raft_consensus.cc:2749] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 08:57:08.023036 26629 leader_election.cc:290] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 3 election: Requested vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105)
I20260321 08:57:08.023144 26634 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:57:08.031582 26631 raft_consensus.cc:2749] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 08:57:08.026255 26632 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
W20260321 08:57:08.028424 26325 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 08:57:08.035948 26632 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 5 election: Requested vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:08.041569 26634 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 08:57:08.048528 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
W20260321 08:57:08.052124 26004 leader_election.cc:336] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111)
I20260321 08:57:08.052753 26004 leader_election.cc:304] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74; no voters: 5c9e21016d934beab3beb647aa48006b, 9af081622eaa4661a29feb3bf5d18cc8
I20260321 08:57:08.052966 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:08.053721 26362 raft_consensus.cc:2393] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate de2947147ecb4d6eafa4aa039126f25b in current term 5: Already voted for candidate d119b009c5744457b0555896e263fa35 in this term.
I20260321 08:57:08.053596 26627 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 08:57:08.061622 26436 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 5.
I20260321 08:57:08.062719 26241 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35
I20260321 08:57:08.063828 26634 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader election won for term 5
I20260321 08:57:08.064322 26634 raft_consensus.cc:697] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 5 LEADER]: Becoming Leader. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:08.065352 26634 consensus_queue.cc:237] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
W20260321 08:57:08.069499 26134 leader_election.cc:336] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111)
I20260321 08:57:08.069944 26134 leader_election.cc:304] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9af081622eaa4661a29feb3bf5d18cc8; no voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 5c9e21016d934beab3beb647aa48006b
I20260321 08:57:08.070559 26629 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 08:57:08.159590 26362 raft_consensus.cc:1275] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Refusing update from remote peer de2947147ecb4d6eafa4aa039126f25b: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 08:57:08.161106 26634 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 08:57:08.168115 26631 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:08.168588 26631 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:08.170531 26631 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:08.174453 26440 raft_consensus.cc:1275] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Refusing update from remote peer de2947147ecb4d6eafa4aa039126f25b: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 08:57:08.176237 26632 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 08:57:08.191951 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:08.192574 26436 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 4.
I20260321 08:57:08.193601 26325 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters:
I20260321 08:57:08.194545 26631 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 08:57:08.194905 26631 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:08.195217 26631 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:08.195362 26622 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:08.195947 26622 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:08.197870 26622 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:08.199033 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:08.200683 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:08.201262 26283 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 4.
I20260321 08:57:08.201411 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:08.203018 26404 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:08.204259 26622 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 08:57:08.204668 26622 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:08.205941 26622 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:08.235423 26622 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:08.241107 26622 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 5 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:08.242611 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:08.243242 26282 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:08.248250 26634 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:08.249423 26622 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:08.249890 26634 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:08.250224 26622 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:08.254551 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:08.254745 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:08.255077 26634 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:08.255337 26362 raft_consensus.cc:2393] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de2947147ecb4d6eafa4aa039126f25b in current term 4: Already voted for candidate d119b009c5744457b0555896e263fa35 in this term.
I20260321 08:57:08.255371 26436 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 4.
I20260321 08:57:08.255749 26282 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 5.
I20260321 08:57:08.257839 26404 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:08.258431 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:08.258733 26621 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Leader election won for term 5
I20260321 08:57:08.259495 26622 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:08.259033 26362 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 4.
I20260321 08:57:08.256776 26241 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:08.262384 26621 raft_consensus.cc:697] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 5 LEADER]: Becoming Leader. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:08.262909 26405 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters:
I20260321 08:57:08.249130 26360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:08.265849 26647 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 08:57:08.266156 26647 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:08.266407 26647 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:08.263487 26621 consensus_queue.cc:237] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:08.275321 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:08.265038 26634 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 08:57:08.276232 26634 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Starting leader election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:08.276587 26634 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:57:08.264793 26631 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:08.279774 26361 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd3e8a46765d4e07b3d1087aca0be2be in current term 5: Already voted for candidate d119b009c5744457b0555896e263fa35 in this term.
I20260321 08:57:08.282829 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:08.282789 26631 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 5 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:08.283456 26283 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 5: Already voted for candidate dd3e8a46765d4e07b3d1087aca0be2be in this term.
I20260321 08:57:08.283576 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:08.285344 26325 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:08.286473 26631 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 08:57:08.289954 26647 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:08.291836 26647 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 5 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:08.291692 26634 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:08.292898 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:08.298209 26361 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:08.298924 26629 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:08.299371 26629 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:08.300994 26629 leader_election.cc:290] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105)
I20260321 08:57:08.301450 26634 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 4 election: Requested vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:08.301748 26360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:08.302345 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 4 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
I20260321 08:57:08.302942 26026 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9af081622eaa4661a29feb3bf5d18cc8 in term 3.
I20260321 08:57:08.304421 26130 leader_election.cc:304] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8; no voters:
I20260321 08:57:08.305303 26282 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 4.
I20260321 08:57:08.307189 26361 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 5.
I20260321 08:57:08.309667 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:08.310128 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:08.310371 26440 raft_consensus.cc:2368] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate de2947147ecb4d6eafa4aa039126f25b for earlier term 4. Current term is 5.
I20260321 08:57:08.310257 26405 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters:
I20260321 08:57:08.311611 26241 leader_election.cc:400] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 4 election: Vote denied by peer dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993) with higher term. Message: Invalid argument: T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate de2947147ecb4d6eafa4aa039126f25b for earlier term 4. Current term is 5.
W20260321 08:57:08.311942 26134 leader_election.cc:336] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111)
I20260321 08:57:08.312101 26647 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Leader election won for term 5
I20260321 08:57:08.312907 26629 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 08:57:08.313283 26629 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:08.313601 26629 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:57:08.315315 26647 raft_consensus.cc:697] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 5 LEADER]: Becoming Leader. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:08.316160 26647 consensus_queue.cc:237] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:08.311951 26241 leader_election.cc:403] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260321 08:57:08.316133 26282 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:08.319921 26629 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:08.324117 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 4 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74"
I20260321 08:57:08.324646 26026 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 3 FOLLOWER]: Advancing to term 4
I20260321 08:57:08.325882 26282 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 5.
I20260321 08:57:08.326747 26634 raft_consensus.cc:2749] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993) with higher term. Message: Invalid argument: T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate de2947147ecb4d6eafa4aa039126f25b for earlier term 4. Current term is 5.
I20260321 08:57:08.332255 26026 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9af081622eaa4661a29feb3bf5d18cc8 in term 4.
I20260321 08:57:08.335675 26130 leader_election.cc:304] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8; no voters:
I20260321 08:57:08.341754 26629 leader_election.cc:290] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 4 election: Requested vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105)
I20260321 08:57:08.344347 26629 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 4 FOLLOWER]: Leader election won for term 4
W20260321 08:57:08.345731 26134 leader_election.cc:336] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111)
I20260321 08:57:08.346758 26629 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 4 LEADER]: Becoming Leader. State: Replica: 9af081622eaa4661a29feb3bf5d18cc8, State: Running, Role: LEADER
I20260321 08:57:08.347786 26629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:08.357241 26361 raft_consensus.cc:1275] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Refusing update from remote peer dd3e8a46765d4e07b3d1087aca0be2be: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 08:57:08.358440 26647 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 08:57:08.363046 26651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af081622eaa4661a29feb3bf5d18cc8. Latest consensus state: current_term: 4 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:08.364066 26651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:08.365698 26654 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:08.378594 26654 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8]
I20260321 08:57:08.379210 26282 raft_consensus.cc:1275] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Refusing update from remote peer dd3e8a46765d4e07b3d1087aca0be2be: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 08:57:08.380602 26622 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 08:57:08.401137 26654 tablet_loader.cc:96] loaded metadata for tablet 3effe2075d034127a1ac31b507b8119a (table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:08.402623 26654 tablet_loader.cc:96] loaded metadata for tablet 51dc0b437c8c49aea324853455672c6c (table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:08.404016 26654 tablet_loader.cc:96] loaded metadata for tablet b4eff2c92f994bb2a318dfe6097efacb (table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:08.404558 26361 raft_consensus.cc:1275] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Refusing update from remote peer dd3e8a46765d4e07b3d1087aca0be2be: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260321 08:57:08.405642 26654 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:08.405655 26622 consensus_queue.cc:1048] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 08:57:08.412317 26654 catalog_manager.cc:1269] Loaded cluster ID: 9774d2365c48473a92ff616a29d5cfc9
I20260321 08:57:08.412640 26654 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:08.418015 26654 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:08.423283 26654 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8: Loaded TSK: 0
I20260321 08:57:08.424470 26654 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:08.429232 26282 raft_consensus.cc:1275] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Refusing update from remote peer dd3e8a46765d4e07b3d1087aca0be2be: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260321 08:57:08.432821 26647 consensus_queue.cc:1048] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 08:57:08.435935 26147 catalog_manager.cc:5671] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: HEALTHY } } }
I20260321 08:57:08.453444 26026 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 4 FOLLOWER]: Refusing update from remote peer 9af081622eaa4661a29feb3bf5d18cc8: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 4 index: 19. (index mismatch)
I20260321 08:57:08.455044 26651 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260321 08:57:08.455907 26134 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 -> Peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Couldn't send request to peer 5c9e21016d934beab3beb647aa48006b. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:08.461490 26627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af081622eaa4661a29feb3bf5d18cc8. Latest consensus state: current_term: 4 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:08.462280 26627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:08.466223 26651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:08.466962 26651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:08.468014 26627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:08.468698 26627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:08.475334 26465 heartbeater.cc:499] Master 127.24.0.188:45371 was elected leader, sending a full tablet report...
I20260321 08:57:08.480175 26309 heartbeater.cc:499] Master 127.24.0.188:45371 was elected leader, sending a full tablet report...
I20260321 08:57:08.481473 26144 catalog_manager.cc:5671] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be reported cstate change: term changed from 3 to 5, leader changed from d119b009c5744457b0555896e263fa35 (127.24.0.130) to dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131). New cstate: current_term: 5 leader_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: HEALTHY } } }
I20260321 08:57:08.483309 26144 catalog_manager.cc:5671] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be reported cstate change: term changed from 4 to 5, leader changed from dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131) to de2947147ecb4d6eafa4aa039126f25b (127.24.0.129). New cstate: current_term: 5 leader_uuid: "de2947147ecb4d6eafa4aa039126f25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } } }
I20260321 08:57:08.513576 26144 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8: NewTableName [id=aec8b423bf774e35beabf2b480fcfda8] alter complete (version 1)
I20260321 08:57:08.578431 26387 heartbeater.cc:499] Master 127.24.0.188:45371 was elected leader, sending a full tablet report...
W20260321 08:57:08.840947 26461 debug-util.cc:398] Leaking SignalData structure 0x7b0800216fe0 after lost signal to thread 24581
W20260321 08:57:08.842198 26461 debug-util.cc:398] Leaking SignalData structure 0x7b0800217740 after lost signal to thread 26051
W20260321 08:57:08.843293 26461 debug-util.cc:398] Leaking SignalData structure 0x7b0800217f60 after lost signal to thread 26181
W20260321 08:57:08.845036 26461 debug-util.cc:398] Leaking SignalData structure 0x7b080022d780 after lost signal to thread 26308
W20260321 08:57:08.846287 26461 debug-util.cc:398] Leaking SignalData structure 0x7b080015e7e0 after lost signal to thread 26386
W20260321 08:57:08.847538 26461 debug-util.cc:398] Leaking SignalData structure 0x7b0800104600 after lost signal to thread 26464
W20260321 08:57:08.957966 26134 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 -> Peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Couldn't send request to peer 5c9e21016d934beab3beb647aa48006b. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260321 08:57:08.999394 24578 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 26113)
W20260321 08:57:09.478111 26134 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 -> Peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Couldn't send request to peer 5c9e21016d934beab3beb647aa48006b. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260321 08:57:09.973707 26134 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 -> Peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Couldn't send request to peer 5c9e21016d934beab3beb647aa48006b. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260321 08:57:09.999912 24578 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 26113)
W20260321 08:57:10.467936 26134 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 -> Peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Couldn't send request to peer 5c9e21016d934beab3beb647aa48006b. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260321 08:57:11.000382 24578 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 26113)
W20260321 08:57:11.064888 26134 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 -> Peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Couldn't send request to peer 5c9e21016d934beab3beb647aa48006b. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260321 08:57:11.569096 26134 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 -> Peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Couldn't send request to peer 5c9e21016d934beab3beb647aa48006b. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260321 08:57:12.001238 24578 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 26113)
W20260321 08:57:12.037465 26134 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 -> Peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Couldn't send request to peer 5c9e21016d934beab3beb647aa48006b. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260321 08:57:12.049700 24578 master.cc:584] Master@127.24.0.189:38105 shutdown complete.
W20260321 08:57:12.151767 26134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20260321 08:57:12.272174 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:12.283242 26667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:12.286067 26668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:12.288900 24578 server_base.cc:1061] running on GCE node
W20260321 08:57:12.289335 26670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:12.291258 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:12.291491 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:12.291677 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083432291662 us; error 0 us; skew 500 ppm
I20260321 08:57:12.292273 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:12.300891 24578 webserver.cc:492] Webserver started at http://127.24.0.189:35349/ using document root <none> and password file <none>
I20260321 08:57:12.301601 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:12.301864 24578 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260321 08:57:12.306248 24578 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260321 08:57:12.309772 26675 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:12.310583 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260321 08:57:12.310935 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-1-root
uuid: "5c9e21016d934beab3beb647aa48006b"
format_stamp: "Formatted at 2026-03-21 08:57:01 on dist-test-slave-p54g"
I20260321 08:57:12.311470 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:12.327641 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:12.328842 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:12.390421 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.189:38105
I20260321 08:57:12.390504 26726 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.189:38105 every 8 connection(s)
I20260321 08:57:12.400485 26727 sys_catalog.cc:263] Verifying existing consensus state
I20260321 08:57:12.405470 26727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: Bootstrap starting.
I20260321 08:57:12.477906 26388 heartbeater.cc:344] Connected to a master server at 127.24.0.189:38105
I20260321 08:57:12.481371 26727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=7 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 08:57:12.481467 26310 heartbeater.cc:344] Connected to a master server at 127.24.0.189:38105
I20260321 08:57:12.482564 26727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: Bootstrap complete.
I20260321 08:57:12.486208 26727 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:12.486897 26727 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9e21016d934beab3beb647aa48006b, State: Initialized, Role: FOLLOWER
I20260321 08:57:12.487701 26727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:12.510437 26732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:12.511339 26732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:12.512241 26727 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:12.512858 26702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 2 FOLLOWER]: Advancing to term 4
I20260321 08:57:12.513392 26466 heartbeater.cc:344] Connected to a master server at 127.24.0.189:38105
I20260321 08:57:12.579746 26732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af081622eaa4661a29feb3bf5d18cc8. Latest consensus state: current_term: 4 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:12.580083 26743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:12.580861 26743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:12.580518 26732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:12.586676 26750 catalog_manager.cc:1269] Loaded cluster ID: 9774d2365c48473a92ff616a29d5cfc9
I20260321 08:57:12.586905 26750 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: loading cluster ID for follower catalog manager: success
I20260321 08:57:12.591395 26750 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: acquiring CA information for follower catalog manager: success
I20260321 08:57:12.595643 26750 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260321 08:57:12.607599 26629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:12.608635 26629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:12.612355 26691 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" instance_seqno: 1774083422231592) as {username='slave'} at 127.0.0.1:53120; Asking this server to re-register.
I20260321 08:57:12.613982 26310 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:12.615386 26692 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d119b009c5744457b0555896e263fa35" instance_seqno: 1774083422369585) as {username='slave'} at 127.0.0.1:53092; Asking this server to re-register.
I20260321 08:57:12.617002 26388 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:12.617377 26692 ts_manager.cc:194] Registered new tserver with Master: de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365)
I20260321 08:57:12.621255 26692 ts_manager.cc:194] Registered new tserver with Master: d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:12.623123 26691 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" instance_seqno: 1774083422518745) as {username='slave'} at 127.0.0.1:53126; Asking this server to re-register.
I20260321 08:57:12.626664 26466 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:12.636956 26692 ts_manager.cc:194] Registered new tserver with Master: dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:12.637070 24578 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.241956822s
I20260321 08:57:18.697492 26761 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:18.698570 26761 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:18.701524 26761 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:18.725947 26765 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:18.727617 26764 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:18.728447 26764 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:18.731616 26764 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:18.726663 26765 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:18.736799 26763 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9af081622eaa4661a29feb3bf5d18cc8)
I20260321 08:57:18.737051 26765 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:18.737465 26763 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:18.740520 26763 leader_election.cc:290] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:18.816552 26765 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:18.816438 26769 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9af081622eaa4661a29feb3bf5d18cc8)
I20260321 08:57:18.817370 26765 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:18.817584 26769 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:18.819376 26765 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:18.820235 26769 leader_election.cc:290] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
W20260321 08:57:18.956235 26676 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.24.0.190:45875 for RPC RequestConsensusVote timed out after 0.213s (ON_OUTBOUND_QUEUE) [suppressed 2 similar messages]
I20260321 08:57:19.258149 26774 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:19.258957 26774 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
W20260321 08:57:19.260504 26327 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.000s sys 0.013s
W20260321 08:57:19.260751 26328 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.704s user 0.007s sys 0.012s
W20260321 08:57:19.261283 26328 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.705s user 0.007s sys 0.013s
W20260321 08:57:19.265509 26327 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s user 0.000s sys 0.014s
I20260321 08:57:19.268003 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:19.268865 26361 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 5.
W20260321 08:57:19.272163 26155 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49802 (request call id 0) took 471 ms (client timeout 213 ms). Trace:
W20260321 08:57:19.272630 26155 rpcz_store.cc:269] 0321 08:57:18.800337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:19.271769 (+471432us) service_pool.cc:212] Skipping call since client already timed out
0321 08:57:19.271947 (+ 178us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260321 08:57:19.275713 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:19.279613 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:19.280272 26282 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 5.
I20260321 08:57:19.283789 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:19.284363 26282 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 5.
I20260321 08:57:19.286561 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:19.291381 26774 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:19.293758 26774 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:19.295452 26774 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:19.313869 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:19.320241 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:19.332931 26774 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
W20260321 08:57:19.333812 26404 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.129:38365, user_credentials={real_user=slave}} blocked reactor thread for 51366.5us
I20260321 08:57:19.334281 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
W20260321 08:57:19.337193 26405 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.24.0.130:41799, user_credentials={real_user=slave}} blocked reactor thread for 44551.1us
W20260321 08:57:19.340420 26676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.190:45875, user_credentials={real_user=slave}} blocked reactor thread for 384298us
I20260321 08:57:19.342499 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:19.344362 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:19.346343 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:19.346902 26361 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 5.
I20260321 08:57:19.347124 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:19.354534 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:19.358253 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
W20260321 08:57:19.359192 26326 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.129:38365, user_credentials={real_user=slave}} blocked reactor thread for 357365us
I20260321 08:57:19.359710 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:19.360478 26702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b" is_pre_election: true
I20260321 08:57:19.360527 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:19.362778 26405 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters: de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:19.362835 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 5 candidate_status { last_received { term: 4 index: 21 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
I20260321 08:57:19.363920 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:19.364490 26783 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 08:57:19.365126 26676 leader_election.cc:304] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9e21016d934beab3beb647aa48006b; no voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8
I20260321 08:57:19.365296 26783 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Starting leader election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:19.362879 26004 leader_election.cc:304] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74; no voters: 5c9e21016d934beab3beb647aa48006b, 9af081622eaa4661a29feb3bf5d18cc8
I20260321 08:57:19.365754 26783 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:19.366523 26784 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 08:57:19.367062 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:19.368489 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:19.370935 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:19.372241 26326 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:19.375232 26783 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:19.377288 26787 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 08:57:19.377661 26244 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:19.379598 26780 raft_consensus.cc:2749] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 08:57:19.380446 26244 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:19.381098 26774 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 08:57:19.382416 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:19.382891 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:19.384477 26326 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:19.385015 26787 raft_consensus.cc:2749] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 08:57:19.387297 26283 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:19.387740 26786 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 08:57:19.388803 26440 raft_consensus.cc:1240] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Rejecting Update request from peer de2947147ecb4d6eafa4aa039126f25b for earlier term 5. Current term is 6. Ops: []
I20260321 08:57:19.389791 26325 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:19.390342 26774 consensus_queue.cc:1059] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260321 08:57:19.390921 26787 raft_consensus.cc:2749] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 08:57:19.391149 26774 raft_consensus.cc:3055] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 5 LEADER]: Stepping down as leader of term 5
I20260321 08:57:19.391505 26774 raft_consensus.cc:740] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:19.392387 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:19.388715 26783 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 6 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:19.392191 26774 consensus_queue.cc:260] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:19.393800 26405 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be; no voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:19.393815 26774 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:19.394724 26785 raft_consensus.cc:2749] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 08:57:19.586375 26774 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:19.586808 26774 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:19.588383 26774 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:19.589147 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:19.589037 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:19.589763 26440 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 6.
I20260321 08:57:19.589898 26362 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 5.
I20260321 08:57:19.590669 26241 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:19.591337 26774 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 08:57:19.591746 26774 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:19.592087 26774 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:19.599352 26774 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:19.601281 26774 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 7 election: Requested vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:19.602113 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:19.602643 26362 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Advancing to term 7
I20260321 08:57:19.603693 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:19.604300 26440 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:19.610793 26440 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 7.
I20260321 08:57:19.610978 26362 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 7.
I20260321 08:57:19.611567 26241 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:19.612352 26774 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Leader election won for term 7
I20260321 08:57:19.612790 26774 raft_consensus.cc:697] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 7 LEADER]: Becoming Leader. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:19.614190 26774 consensus_queue.cc:237] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:19.624006 26144 catalog_manager.cc:5671] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "de2947147ecb4d6eafa4aa039126f25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: UNKNOWN } } }
I20260321 08:57:19.696717 26440 raft_consensus.cc:1275] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Refusing update from remote peer de2947147ecb4d6eafa4aa039126f25b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260321 08:57:19.697850 26774 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260321 08:57:19.712188 26362 raft_consensus.cc:1275] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Refusing update from remote peer de2947147ecb4d6eafa4aa039126f25b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260321 08:57:19.713722 26776 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260321 08:57:20.282276 26461 debug-util.cc:398] Leaking SignalData structure 0x7b08001670a0 after lost signal to thread 24581
W20260321 08:57:20.283121 26461 debug-util.cc:398] Leaking SignalData structure 0x7b08000f51c0 after lost signal to thread 26051
W20260321 08:57:20.283754 26461 debug-util.cc:398] Leaking SignalData structure 0x7b0800227220 after lost signal to thread 26181
W20260321 08:57:20.284642 26461 debug-util.cc:398] Leaking SignalData structure 0x7b0800217400 after lost signal to thread 26308
W20260321 08:57:20.285392 26461 debug-util.cc:398] Leaking SignalData structure 0x7b080013cfa0 after lost signal to thread 26386
W20260321 08:57:20.286118 26461 debug-util.cc:398] Leaking SignalData structure 0x7b0800241840 after lost signal to thread 26464
W20260321 08:57:20.286832 26461 debug-util.cc:398] Leaking SignalData structure 0x7b080009c3e0 after lost signal to thread 26726
I20260321 08:57:27.114670 26816 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9af081622eaa4661a29feb3bf5d18cc8)
I20260321 08:57:27.115412 26816 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:27.141290 26702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b" is_pre_election: true
I20260321 08:57:27.141803 26816 leader_election.cc:290] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:27.141968 26702 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e553cc2497492a9aa4dd5cb1d3ae74 in term 4.
I20260321 08:57:27.126477 26815 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:27.143422 26004 leader_election.cc:304] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 5c9e21016d934beab3beb647aa48006b; no voters:
I20260321 08:57:27.143711 26815 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:27.146090 26815 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:27.220423 26774 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:27.221022 26774 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:27.223341 26774 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:27.309379 26774 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:27.309949 26774 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:27.310351 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:27.331980 26774 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:27.347394 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:27.374083 26816 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 08:57:27.372025 26813 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9af081622eaa4661a29feb3bf5d18cc8)
I20260321 08:57:27.400525 26817 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:27.401010 26816 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9af081622eaa4661a29feb3bf5d18cc8)
I20260321 08:57:27.418387 26360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:27.401391 26817 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:27.417723 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:27.418919 26816 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:27.419315 26360 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 5.
I20260321 08:57:27.419471 26361 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 5.
I20260321 08:57:27.421703 26244 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:27.400345 26813 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:27.422559 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:27.422551 26244 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:27.424901 26813 leader_election.cc:290] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
W20260321 08:57:27.547075 26241 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.24.0.131:36993 timed out after 0.256s (SENT) [suppressed 9 similar messages]
I20260321 08:57:27.420614 26817 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
W20260321 08:57:28.009526 26244 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.018s sys 0.057s
W20260321 08:57:28.010941 26244 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.587s user 0.018s sys 0.058s
W20260321 08:57:28.011919 26244 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.130:41799, user_credentials={real_user=slave}} blocked reactor thread for 589390us
I20260321 08:57:28.008641 26817 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:28.062721 26818 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260321 08:57:28.068787 26816 consensus_meta.cc:287] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74: Time spent flushing consensus metadata: real 0.649s user 0.003s sys 0.005s
I20260321 08:57:28.091593 26818 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:28.094581 26818 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:28.093132 26816 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:28.098210 26816 leader_election.cc:290] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 5 election: Requested vote from peers 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:28.124446 26817 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:28.140276 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:28.140985 26284 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 5.
I20260321 08:57:28.142247 26326 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:28.146240 26817 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:28.146807 26817 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 08:57:28.147121 26817 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:28.147467 26817 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:28.168442 26817 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:28.175302 26817 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:28.176004 26817 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:28.176507 26817 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:28.178800 26817 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:28.273818 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:28.278460 26818 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:28.280726 26818 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 election: Requested vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:28.281287 26818 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 08:57:28.281662 26818 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Starting leader election (detected failure of leader dd3e8a46765d4e07b3d1087aca0be2be)
I20260321 08:57:28.282014 26818 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:28.285384 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:28.288054 26818 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:28.289983 26818 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 election: Requested vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
W20260321 08:57:28.016492 26131 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.006s sys 0.014s
W20260321 08:57:28.328773 26131 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.818s user 0.006s sys 0.015s
W20260321 08:57:28.331477 26241 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.131:36993, user_credentials={real_user=slave}} blocked reactor thread for 784228us
I20260321 08:57:28.331707 26026 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 5 FOLLOWER]: Rejecting Update request from peer 9af081622eaa4661a29feb3bf5d18cc8 for earlier term 4. Current term is 5. Ops: []
W20260321 08:57:28.327741 24583 debug-util.cc:398] Leaking SignalData structure 0x7b080002b640 after lost signal to thread 26405
I20260321 08:57:28.333002 26027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
W20260321 08:57:28.334080 24583 kernel_stack_watchdog.cc:198] Thread 26405 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 278ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20260321 08:57:28.336135 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:28.341454 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8"
I20260321 08:57:28.342803 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:28.345813 26362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:28.346592 26362 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:28.346894 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:28.347971 26282 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 6: Already voted for candidate de2947147ecb4d6eafa4aa039126f25b in this term.
I20260321 08:57:28.349695 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:28.350912 26284 raft_consensus.cc:2393] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d119b009c5744457b0555896e263fa35 in current term 6: Already voted for candidate de2947147ecb4d6eafa4aa039126f25b in this term.
I20260321 08:57:28.354230 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:28.354887 26676 leader_election.cc:304] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9e21016d934beab3beb647aa48006b; no voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8
I20260321 08:57:28.355680 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:28.357026 26362 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2947147ecb4d6eafa4aa039126f25b in term 6.
I20260321 08:57:28.358966 26244 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters: dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:28.359592 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:28.360708 26701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b"
I20260321 08:57:28.362182 26004 leader_election.cc:304] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74; no voters: 5c9e21016d934beab3beb647aa48006b, 9af081622eaa4661a29feb3bf5d18cc8
I20260321 08:57:28.362628 26774 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Leader election won for term 6
I20260321 08:57:28.363219 26774 raft_consensus.cc:697] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 6 LEADER]: Becoming Leader. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:28.363695 26816 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 08:57:28.364482 26774 consensus_queue.cc:237] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:28.367156 26823 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260321 08:57:28.368043 26405 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.384s user 0.005s sys 0.065s
W20260321 08:57:28.368474 26405 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.384s user 0.005s sys 0.065s
I20260321 08:57:28.372438 26360 raft_consensus.cc:1240] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Rejecting Update request from peer dd3e8a46765d4e07b3d1087aca0be2be for earlier term 5. Current term is 6. Ops: []
I20260321 08:57:28.372537 26362 raft_consensus.cc:1240] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Rejecting Update request from peer dd3e8a46765d4e07b3d1087aca0be2be for earlier term 5. Current term is 6. Ops: []
I20260321 08:57:28.374086 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:28.374507 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:28.374037 26359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:28.375821 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:28.377404 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:28.378822 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:28.379032 26325 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:28.381021 26361 raft_consensus.cc:2393] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate de2947147ecb4d6eafa4aa039126f25b in current term 6: Already voted for candidate d119b009c5744457b0555896e263fa35 in this term.
I20260321 08:57:28.381417 26822 raft_consensus.cc:2749] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 08:57:28.382040 26244 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:28.380894 26325 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:28.383109 26822 raft_consensus.cc:2749] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 08:57:28.385003 26774 raft_consensus.cc:2749] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 08:57:28.386648 26804 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 }, Status: INVALID_TERM, Last received: 4.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260321 08:57:28.387225 26825 consensus_queue.cc:1059] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 08:57:28.388003 26804 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 08:57:28.388367 26804 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9af081622eaa4661a29feb3bf5d18cc8, State: Running, Role: LEADER
I20260321 08:57:28.389482 26804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 4.22, Last appended by leader: 22, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:28.390987 26804 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:28.394444 26825 raft_consensus.cc:3055] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 5 LEADER]: Stepping down as leader of term 5
I20260321 08:57:28.394879 26825 raft_consensus.cc:740] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:28.395989 26825 consensus_queue.cc:260] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:28.397362 26825 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:28.401093 26325 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:28.401968 26822 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 08:57:28.398036 26827 consensus_queue.cc:1059] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260321 08:57:28.406934 26406 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 08:57:28.409641 26702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53104 (request call id 139) took 1036 ms. Trace:
I20260321 08:57:28.412844 26359 raft_consensus.cc:1240] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Rejecting Update request from peer dd3e8a46765d4e07b3d1087aca0be2be for earlier term 5. Current term is 6. Ops: []
I20260321 08:57:28.414323 26825 consensus_queue.cc:1059] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260321 08:57:28.414726 26048 debug-util.cc:398] Leaking SignalData structure 0x7b080003cac0 after lost signal to thread 24581
W20260321 08:57:28.416083 26048 debug-util.cc:398] Leaking SignalData structure 0x7b080004eb00 after lost signal to thread 26051
I20260321 08:57:28.416213 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:28.416157 26826 raft_consensus.cc:3055] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 5 LEADER]: Stepping down as leader of term 5
I20260321 08:57:28.416999 26826 raft_consensus.cc:740] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:28.414216 26702 rpcz_store.cc:276] 0321 08:57:27.373197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 08:57:27.373454 (+ 257us) service_pool.cc:225] Handling call
0321 08:57:27.373717 (+ 263us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 08:57:28.408575 (+1034858us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 08:57:28.408650 (+ 75us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 08:57:28.408692 (+ 42us) raft_consensus.cc:1645] Marking committed up to 22
0321 08:57:28.408747 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 08:57:28.409236 (+ 489us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54004d0154. stack: 00007f8b66b4b413 00007f8b66b4b310 00007f8b66499574 00000000003bff74 00000000003bff09 00007f8b6d4e1951 00007f8b6d5791e1 00007f8b6d577c1e 00007f8b6f57ead9 00007f8b6d020d34 00007f8b6d020cc2 00007f8b6d020bf1 00007f8b6d020b6c 00007f8b6d01fe22 00007f8b6b74c5cc 00007f8b6b74ba06
0321 08:57:28.409269 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 08:57:28.409436 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2379327872}
W20260321 08:57:28.417724 26048 debug-util.cc:398] Leaking SignalData structure 0x7b080014af80 after lost signal to thread 26181
I20260321 08:57:28.417934 26404 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be; no voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b
W20260321 08:57:28.419353 26048 debug-util.cc:398] Leaking SignalData structure 0x7b0800163300 after lost signal to thread 26308
I20260321 08:57:28.420305 26825 raft_consensus.cc:2749] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 08:57:28.421209 26048 debug-util.cc:398] Leaking SignalData structure 0x7b08002371a0 after lost signal to thread 26386
W20260321 08:57:28.422117 26048 debug-util.cc:398] Leaking SignalData structure 0x7b0800106fc0 after lost signal to thread 26464
I20260321 08:57:28.422206 26826 consensus_queue.cc:260] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
W20260321 08:57:28.422930 26048 debug-util.cc:398] Leaking SignalData structure 0x7b080009cb20 after lost signal to thread 26726
I20260321 08:57:28.423364 26826 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:28.470528 26359 raft_consensus.cc:1275] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Refusing update from remote peer de2947147ecb4d6eafa4aa039126f25b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 08:57:28.471616 26774 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260321 08:57:28.489389 26439 raft_consensus.cc:1275] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Refusing update from remote peer de2947147ecb4d6eafa4aa039126f25b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 08:57:28.494583 26774 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260321 08:57:28.584841 26822 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:28.585373 26822 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:28.587491 26822 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:28.588167 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:28.588388 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:28.588801 26284 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 6.
I20260321 08:57:28.588917 26439 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 6.
I20260321 08:57:28.589788 26325 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters:
I20260321 08:57:28.590477 26822 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 08:57:28.590750 26822 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:28.590981 26822 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:28.596964 26822 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:28.598932 26816 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:28.599418 26816 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:28.600236 26822 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 7 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:28.601452 26816 leader_election.cc:290] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:28.601689 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:28.602252 26284 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:28.602706 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 6 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:28.603229 26155 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e553cc2497492a9aa4dd5cb1d3ae74 in term 5.
I20260321 08:57:28.604338 26002 leader_election.cc:304] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8; no voters:
I20260321 08:57:28.604943 26702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 6 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b" is_pre_election: true
I20260321 08:57:28.606707 26816 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 08:57:28.607072 26816 raft_consensus.cc:493] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:28.607375 26816 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:28.609859 26823 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9af081622eaa4661a29feb3bf5d18cc8)
I20260321 08:57:28.610376 26823 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:28.612600 26823 leader_election.cc:290] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:28.613413 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:28.613906 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
I20260321 08:57:28.614265 26816 raft_consensus.cc:515] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:28.615206 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:28.616773 26439 raft_consensus.cc:3060] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:28.617169 26155 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9e21016d934beab3beb647aa48006b in term 5.
I20260321 08:57:28.619107 26026 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c9e21016d934beab3beb647aa48006b for earlier term 5. Current term is 6.
I20260321 08:57:28.619505 26677 leader_election.cc:304] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9e21016d934beab3beb647aa48006b, 9af081622eaa4661a29feb3bf5d18cc8; no voters:
I20260321 08:57:28.620476 26823 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 08:57:28.620844 26823 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9af081622eaa4661a29feb3bf5d18cc8)
I20260321 08:57:28.621182 26823 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 4 FOLLOWER]: Advancing to term 5
I20260321 08:57:28.622522 26676 leader_election.cc:400] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 5 pre-election: Vote denied by peer 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c9e21016d934beab3beb647aa48006b for earlier term 5. Current term is 6.
I20260321 08:57:28.616043 26284 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 7.
I20260321 08:57:28.626250 26326 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:28.627725 26822 raft_consensus.cc:2804] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Leader election won for term 7
I20260321 08:57:28.628301 26822 raft_consensus.cc:697] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 7 LEADER]: Becoming Leader. State: Replica: d119b009c5744457b0555896e263fa35, State: Running, Role: LEADER
I20260321 08:57:28.629285 26822 consensus_queue.cc:237] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:28.629945 26816 leader_election.cc:290] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 6 election: Requested vote from peers 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:28.631551 26702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 6 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b"
I20260321 08:57:28.632272 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" candidate_term: 6 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8"
I20260321 08:57:28.632838 26155 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:28.633010 26439 raft_consensus.cc:2468] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 7.
I20260321 08:57:28.638878 26823 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:28.640094 26702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 5 FOLLOWER]: Advancing to term 6
I20260321 08:57:28.640504 26155 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e553cc2497492a9aa4dd5cb1d3ae74 in term 6.
I20260321 08:57:28.641614 26002 leader_election.cc:304] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8; no voters:
I20260321 08:57:28.642457 26816 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 6 FOLLOWER]: Leader election won for term 6
I20260321 08:57:28.642899 26816 raft_consensus.cc:697] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 6 LEADER]: Becoming Leader. State: Replica: 42e553cc2497492a9aa4dd5cb1d3ae74, State: Running, Role: LEADER
I20260321 08:57:28.643982 26816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 4.22, Last appended by leader: 22, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:28.648501 26702 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e553cc2497492a9aa4dd5cb1d3ae74 in term 6.
I20260321 08:57:28.656306 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74"
I20260321 08:57:28.658393 26835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42e553cc2497492a9aa4dd5cb1d3ae74. Latest consensus state: current_term: 6 leader_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:28.659222 26835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:28.659461 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 5 candidate_status { last_received { term: 4 index: 22 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8"
I20260321 08:57:28.660207 26155 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9e21016d934beab3beb647aa48006b for earlier term 5. Current term is 6.
I20260321 08:57:28.664160 26677 leader_election.cc:400] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 5 election: Vote denied by peer 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9e21016d934beab3beb647aa48006b for earlier term 5. Current term is 6.
I20260321 08:57:28.664546 26677 leader_election.cc:403] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260321 08:57:28.669101 26823 leader_election.cc:290] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 5 election: Requested vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:28.669376 26839 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c9e21016d934beab3beb647aa48006b for earlier term 5. Current term is 6.
I20260321 08:57:28.671772 26838 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:28.683211 26838 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8]
I20260321 08:57:28.692644 26838 tablet_loader.cc:96] loaded metadata for tablet 3effe2075d034127a1ac31b507b8119a (table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:28.693998 26838 tablet_loader.cc:96] loaded metadata for tablet 51dc0b437c8c49aea324853455672c6c (table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:28.695250 26838 tablet_loader.cc:96] loaded metadata for tablet b4eff2c92f994bb2a318dfe6097efacb (table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:28.696573 26838 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:28.700754 26838 catalog_manager.cc:1269] Loaded cluster ID: 9774d2365c48473a92ff616a29d5cfc9
I20260321 08:57:28.700969 26838 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:28.704738 26838 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:28.707621 26838 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74: Loaded TSK: 0
I20260321 08:57:28.708361 26838 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:28.739123 26439 raft_consensus.cc:1275] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Refusing update from remote peer d119b009c5744457b0555896e263fa35: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260321 08:57:28.740391 26822 consensus_queue.cc:1048] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 08:57:28.743785 26468 heartbeater.cc:499] Master 127.24.0.190:45875 was elected leader, sending a full tablet report...
I20260321 08:57:28.744416 26284 raft_consensus.cc:1275] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Refusing update from remote peer d119b009c5744457b0555896e263fa35: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260321 08:57:28.745931 26822 consensus_queue.cc:1048] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 08:57:28.756814 26017 catalog_manager.cc:5671] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be reported cstate change: term changed from 5 to 6, leader changed from dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131) to de2947147ecb4d6eafa4aa039126f25b (127.24.0.129). New cstate: current_term: 6 leader_uuid: "de2947147ecb4d6eafa4aa039126f25b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } } }
I20260321 08:57:28.760823 26702 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 6 FOLLOWER]: Refusing update from remote peer 42e553cc2497492a9aa4dd5cb1d3ae74: Log matching property violated. Preceding OpId in replica: term: 4 index: 22. Preceding OpId from leader: term: 6 index: 23. (index mismatch)
I20260321 08:57:28.762404 26835 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260321 08:57:28.758343 26017 catalog_manager.cc:5671] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be reported cstate change: term changed from 5 to 7, leader changed from dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131) to d119b009c5744457b0555896e263fa35 (127.24.0.130). New cstate: current_term: 7 leader_uuid: "d119b009c5744457b0555896e263fa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } } }
I20260321 08:57:28.775019 26155 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Refusing update from remote peer 42e553cc2497492a9aa4dd5cb1d3ae74: Log matching property violated. Preceding OpId in replica: term: 4 index: 22. Preceding OpId from leader: term: 6 index: 23. (index mismatch)
I20260321 08:57:28.777015 26835 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260321 08:57:28.790623 26839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42e553cc2497492a9aa4dd5cb1d3ae74. Latest consensus state: current_term: 6 leader_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:28.791324 26839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:28.793830 26835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:28.795647 26835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:28.809424 26804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42e553cc2497492a9aa4dd5cb1d3ae74. Latest consensus state: current_term: 6 leader_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:28.810516 26804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:28.809326 26839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:28.811286 26839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:28.814905 26816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:28.815629 26816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:28.829721 26804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:28.830451 26804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:28.831787 26311 heartbeater.cc:499] Master 127.24.0.190:45875 was elected leader, sending a full tablet report...
I20260321 08:57:28.833639 26389 heartbeater.cc:499] Master 127.24.0.190:45875 was elected leader, sending a full tablet report...
I20260321 08:57:30.683596 26853 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:30.684154 26853 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:30.688850 26854 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42e553cc2497492a9aa4dd5cb1d3ae74)
I20260321 08:57:30.689524 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:30.689432 26854 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:30.691603 26854 leader_election.cc:290] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 9af081622eaa4661a29feb3bf5d18cc8 (127.24.0.188:45371)
I20260321 08:57:30.691505 26853 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:30.711719 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:30.713099 26244 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:30.713714 26855 raft_consensus.cc:2749] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 08:57:30.772651 26858 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42e553cc2497492a9aa4dd5cb1d3ae74)
I20260321 08:57:30.773221 26858 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:30.776036 26858 leader_election.cc:290] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105)
I20260321 08:57:30.777308 26702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b" is_pre_election: true
I20260321 08:57:30.834698 26155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:30.854009 26027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
I20260321 08:57:30.854239 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
I20260321 08:57:30.857683 26130 leader_election.cc:304] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9af081622eaa4661a29feb3bf5d18cc8; no voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 5c9e21016d934beab3beb647aa48006b
I20260321 08:57:30.858330 26858 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 08:57:30.903867 26859 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:30.904503 26859 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:30.906553 26859 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:30.919699 26857 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:30.920271 26857 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:30.922242 26857 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:30.931352 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:30.932891 26860 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:30.933512 26860 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:30.936051 26860 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:30.949163 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:30.952839 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:31.124507 26858 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 42e553cc2497492a9aa4dd5cb1d3ae74)
I20260321 08:57:31.125161 26858 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:31.127491 26858 leader_election.cc:290] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105)
I20260321 08:57:31.128007 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
I20260321 08:57:31.152025 26856 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:31.152619 26856 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.154645 26856 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d119b009c5744457b0555896e263fa35 (127.24.0.130:41799), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:31.156371 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:31.356621 26436 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:31.356657 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:31.357255 26436 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 6.
I20260321 08:57:31.357824 26702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b" is_pre_election: true
I20260321 08:57:31.358354 26702 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9af081622eaa4661a29feb3bf5d18cc8 in term 6.
I20260321 08:57:31.358428 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:31.359223 26134 leader_election.cc:304] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9e21016d934beab3beb647aa48006b, 9af081622eaa4661a29feb3bf5d18cc8; no voters: 42e553cc2497492a9aa4dd5cb1d3ae74
I20260321 08:57:31.358971 26439 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 7.
I20260321 08:57:31.359941 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" is_pre_election: true
I20260321 08:57:31.360505 26858 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 08:57:31.360877 26858 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 42e553cc2497492a9aa4dd5cb1d3ae74)
I20260321 08:57:31.361202 26858 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:31.361654 26405 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be; no voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:31.365818 26863 raft_consensus.cc:493] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d119b009c5744457b0555896e263fa35)
I20260321 08:57:31.366492 26859 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 08:57:31.366354 26863 raft_consensus.cc:515] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.369918 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:31.370110 26863 leader_election.cc:290] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:31.369114 26858 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:31.372135 26361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:31.373199 26405 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be; no voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:31.373485 26026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74"
I20260321 08:57:31.374286 26863 raft_consensus.cc:2749] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 08:57:31.374626 26325 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.131:36993, user_credentials={real_user=slave}} blocked reactor thread for 226908us
I20260321 08:57:31.376972 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:31.377521 26439 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 7.
I20260321 08:57:31.378598 26325 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters: de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:31.379858 26860 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 08:57:31.380246 26860 raft_consensus.cc:493] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Starting leader election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:31.380584 26860 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Advancing to term 8
I20260321 08:57:31.380864 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:31.381444 26439 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 6.
I20260321 08:57:31.382200 26325 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters: de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:31.382754 26857 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 08:57:31.383044 26857 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Starting leader election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:31.383258 26857 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:31.391060 26858 leader_election.cc:290] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 7 election: Requested vote from peers 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875), 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105)
I20260321 08:57:31.391288 26157 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 7 FOLLOWER]: Rejecting Update request from peer 42e553cc2497492a9aa4dd5cb1d3ae74 for earlier term 6. Current term is 7. Ops: []
I20260321 08:57:31.392635 26702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9af081622eaa4661a29feb3bf5d18cc8" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "5c9e21016d934beab3beb647aa48006b"
I20260321 08:57:31.393167 26702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:31.387807 26860 raft_consensus.cc:515] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.395762 26851 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 }, Status: INVALID_TERM, Last received: 6.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260321 08:57:31.400185 26860 leader_election.cc:290] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 8 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:31.405910 26865 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 6 LEADER]: Stepping down as leader of term 6
I20260321 08:57:31.406692 26865 raft_consensus.cc:740] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 42e553cc2497492a9aa4dd5cb1d3ae74, State: Running, Role: LEADER
I20260321 08:57:31.400099 26359 raft_consensus.cc:1240] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 8 FOLLOWER]: Rejecting Update request from peer de2947147ecb4d6eafa4aa039126f25b for earlier term 7. Current term is 8. Ops: []
I20260321 08:57:31.410187 26856 consensus_queue.cc:1059] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 08:57:31.411140 26856 raft_consensus.cc:3055] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 7 LEADER]: Stepping down as leader of term 7
I20260321 08:57:31.411770 26856 raft_consensus.cc:740] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:31.407907 26865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:31.412580 26856 consensus_queue.cc:260] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.413583 26856 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Advancing to term 8
I20260321 08:57:31.415530 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51dc0b437c8c49aea324853455672c6c" candidate_uuid: "de2947147ecb4d6eafa4aa039126f25b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:31.417685 26241 leader_election.cc:304] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:31.418666 26855 raft_consensus.cc:2749] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 08:57:31.419538 26157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c9e21016d934beab3beb647aa48006b" candidate_term: 7 candidate_status { last_received { term: 6 index: 24 } } ignore_live_leader: false dest_uuid: "9af081622eaa4661a29feb3bf5d18cc8" is_pre_election: true
I20260321 08:57:31.420873 26157 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5c9e21016d934beab3beb647aa48006b in current term 7: Already voted for candidate 9af081622eaa4661a29feb3bf5d18cc8 in this term.
I20260321 08:57:31.421725 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:31.423772 26677 leader_election.cc:304] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c9e21016d934beab3beb647aa48006b; no voters: 42e553cc2497492a9aa4dd5cb1d3ae74, 9af081622eaa4661a29feb3bf5d18cc8
I20260321 08:57:31.424744 26702 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9af081622eaa4661a29feb3bf5d18cc8 in term 7.
I20260321 08:57:31.425869 26134 leader_election.cc:304] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5c9e21016d934beab3beb647aa48006b, 9af081622eaa4661a29feb3bf5d18cc8; no voters: 42e553cc2497492a9aa4dd5cb1d3ae74
I20260321 08:57:31.426892 26858 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 7 FOLLOWER]: Leader election won for term 7
I20260321 08:57:31.428619 26857 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.431149 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3effe2075d034127a1ac31b507b8119a" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:31.431170 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:31.437029 26439 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:31.439118 26325 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d119b009c5744457b0555896e263fa35; no voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:31.440361 26822 raft_consensus.cc:2749] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 08:57:31.443924 26865 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:31.444932 26857 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 7 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:31.446689 26858 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 7 LEADER]: Becoming Leader. State: Replica: 9af081622eaa4661a29feb3bf5d18cc8, State: Running, Role: LEADER
I20260321 08:57:31.447805 26858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 6.24, Last appended by leader: 24, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } }
I20260321 08:57:31.454162 26359 raft_consensus.cc:1240] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Rejecting Update request from peer de2947147ecb4d6eafa4aa039126f25b for earlier term 6. Current term is 7. Ops: []
I20260321 08:57:31.455511 26284 raft_consensus.cc:2468] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 8.
I20260321 08:57:31.456475 26832 consensus_queue.cc:1059] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 08:57:31.457445 26832 raft_consensus.cc:3055] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 6 LEADER]: Stepping down as leader of term 6
I20260321 08:57:31.457772 26832 raft_consensus.cc:740] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: de2947147ecb4d6eafa4aa039126f25b, State: Running, Role: LEADER
I20260321 08:57:31.458505 26832 consensus_queue.cc:260] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.459601 26832 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:31.460532 26326 leader_election.cc:304] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters: dd3e8a46765d4e07b3d1087aca0be2be
I20260321 08:57:31.464658 26857 raft_consensus.cc:2804] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 8 FOLLOWER]: Leader election won for term 8
I20260321 08:57:31.465550 26857 raft_consensus.cc:697] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 8 LEADER]: Becoming Leader. State: Replica: d119b009c5744457b0555896e263fa35, State: Running, Role: LEADER
I20260321 08:57:31.466374 26857 consensus_queue.cc:237] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
W20260321 08:57:31.474735 26677 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.24.0.188:45371, user_credentials={real_user=slave}} blocked reactor thread for 51035.4us
I20260321 08:57:31.475708 26866 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 08:57:31.491488 26867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af081622eaa4661a29feb3bf5d18cc8. Latest consensus state: current_term: 7 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:31.492287 26867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:31.527760 26702 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 7 FOLLOWER]: Refusing update from remote peer 9af081622eaa4661a29feb3bf5d18cc8: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
I20260321 08:57:31.529098 26867 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260321 08:57:31.533762 26866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af081622eaa4661a29feb3bf5d18cc8. Latest consensus state: current_term: 7 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:31.534459 26866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:31.546084 26284 raft_consensus.cc:1275] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 8 FOLLOWER]: Refusing update from remote peer d119b009c5744457b0555896e263fa35: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260321 08:57:31.547736 26857 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 08:57:31.559159 26867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:31.559937 26867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:31.560081 26026 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 7 FOLLOWER]: Refusing update from remote peer 9af081622eaa4661a29feb3bf5d18cc8: Log matching property violated. Preceding OpId in replica: term: 6 index: 24. Preceding OpId from leader: term: 7 index: 25. (index mismatch)
I20260321 08:57:31.561533 26858 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260321 08:57:31.569423 26866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:31.570122 26866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:31.589267 26439 raft_consensus.cc:3060] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Advancing to term 8
I20260321 08:57:31.589707 26865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af081622eaa4661a29feb3bf5d18cc8. Latest consensus state: current_term: 7 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:31.590390 26865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:31.589275 26864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:31.591640 26864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [sys.catalog]: This master's current role is: FOLLOWER
I20260321 08:57:31.597704 26439 raft_consensus.cc:1275] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 8 FOLLOWER]: Refusing update from remote peer d119b009c5744457b0555896e263fa35: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260321 08:57:31.599073 26822 consensus_queue.cc:1048] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 08:57:31.618095 26863 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Starting pre-election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:31.618659 26863 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.622675 26863 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:31.625512 26359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35" is_pre_election: true
I20260321 08:57:31.626233 26359 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd3e8a46765d4e07b3d1087aca0be2be in current term 7: Already voted for candidate d119b009c5744457b0555896e263fa35 in this term.
I20260321 08:57:31.632579 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:31.633101 26282 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 7.
I20260321 08:57:31.633920 26404 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35
I20260321 08:57:31.634948 26863 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 08:57:31.635295 26863 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Starting leader election (detected failure of leader de2947147ecb4d6eafa4aa039126f25b)
I20260321 08:57:31.635633 26863 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 6 FOLLOWER]: Advancing to term 7
I20260321 08:57:31.645197 26863 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.647300 26284 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:31.647430 26359 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d119b009c5744457b0555896e263fa35"
I20260321 08:57:31.648272 26359 raft_consensus.cc:2393] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate dd3e8a46765d4e07b3d1087aca0be2be in current term 7: Already voted for candidate d119b009c5744457b0555896e263fa35 in this term.
I20260321 08:57:31.647806 26863 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 7 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), d119b009c5744457b0555896e263fa35 (127.24.0.130:41799)
I20260321 08:57:31.652057 26857 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:31.652549 26857 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.655102 26857 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:31.655226 26858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "9af081622eaa4661a29feb3bf5d18cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e553cc2497492a9aa4dd5cb1d3ae74" member_type: VOTER last_known_addr { host: "127.24.0.190" port: 45875 } } peers { permanent_uuid: "5c9e21016d934beab3beb647aa48006b" member_type: VOTER last_known_addr { host: "127.24.0.189" port: 38105 } } peers { permanent_uuid: "9af081622eaa4661a29feb3bf5d18cc8" member_type: VOTER last_known_addr { host: "127.24.0.188" port: 45371 } } }
I20260321 08:57:31.655908 26858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:31.656904 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b" is_pre_election: true
I20260321 08:57:31.659250 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" is_pre_election: true
I20260321 08:57:31.652751 26284 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3e8a46765d4e07b3d1087aca0be2be in term 7.
I20260321 08:57:31.659834 26440 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 7.
I20260321 08:57:31.660902 26325 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d119b009c5744457b0555896e263fa35, dd3e8a46765d4e07b3d1087aca0be2be; no voters: de2947147ecb4d6eafa4aa039126f25b
I20260321 08:57:31.660965 26404 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dd3e8a46765d4e07b3d1087aca0be2be, de2947147ecb4d6eafa4aa039126f25b; no voters: d119b009c5744457b0555896e263fa35
I20260321 08:57:31.661907 26863 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Leader election won for term 7
I20260321 08:57:31.662072 26857 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 08:57:31.662550 26857 raft_consensus.cc:493] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:31.662933 26857 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Advancing to term 8
I20260321 08:57:31.669184 26857 raft_consensus.cc:515] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.664003 26863 raft_consensus.cc:697] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 7 LEADER]: Becoming Leader. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:31.670992 26863 consensus_queue.cc:237] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.671648 26282 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "de2947147ecb4d6eafa4aa039126f25b"
I20260321 08:57:31.672152 26282 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Advancing to term 8
I20260321 08:57:31.679363 26282 raft_consensus.cc:2468] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d119b009c5744457b0555896e263fa35 in term 8.
I20260321 08:57:31.680435 26326 leader_election.cc:304] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d119b009c5744457b0555896e263fa35, de2947147ecb4d6eafa4aa039126f25b; no voters:
I20260321 08:57:31.681294 26822 raft_consensus.cc:2804] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 8 FOLLOWER]: Leader election won for term 8
I20260321 08:57:31.681766 26822 raft_consensus.cc:697] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 8 LEADER]: Becoming Leader. State: Replica: d119b009c5744457b0555896e263fa35, State: Running, Role: LEADER
I20260321 08:57:31.682610 26822 consensus_queue.cc:237] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.691440 26440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4eff2c92f994bb2a318dfe6097efacb" candidate_uuid: "d119b009c5744457b0555896e263fa35" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dd3e8a46765d4e07b3d1087aca0be2be"
I20260321 08:57:31.693763 26857 leader_election.cc:290] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [CANDIDATE]: Term 8 election: Requested vote from peers de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365), dd3e8a46765d4e07b3d1087aca0be2be (127.24.0.131:36993)
I20260321 08:57:31.765208 26359 raft_consensus.cc:1240] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 8 LEADER]: Rejecting Update request from peer dd3e8a46765d4e07b3d1087aca0be2be for earlier term 7. Current term is 8. Ops: []
I20260321 08:57:31.766254 26863 consensus_queue.cc:1059] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 08:57:31.766796 26863 consensus_queue.cc:1243] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [LEADER]: Peer d119b009c5744457b0555896e263fa35 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260321 08:57:31.767166 26859 raft_consensus.cc:3055] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 7 LEADER]: Stepping down as leader of term 7
I20260321 08:57:31.767431 26859 raft_consensus.cc:740] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 7 LEADER]: Becoming Follower/Learner. State: Replica: dd3e8a46765d4e07b3d1087aca0be2be, State: Running, Role: LEADER
I20260321 08:57:31.768088 26859 consensus_queue.cc:260] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } }
I20260321 08:57:31.768978 26859 raft_consensus.cc:3060] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Advancing to term 8
I20260321 08:57:31.775018 26440 raft_consensus.cc:1275] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 8 FOLLOWER]: Refusing update from remote peer d119b009c5744457b0555896e263fa35: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260321 08:57:31.775277 26440 pending_rounds.cc:85] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be: Aborting all ops after (but not including) 7
I20260321 08:57:31.775447 26440 pending_rounds.cc:107] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260321 08:57:31.775636 26440 raft_consensus.cc:2889] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 08:57:31.778815 26857 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260321 08:57:31.792657 26178 debug-util.cc:398] Leaking SignalData structure 0x7b08000bf2c0 after lost signal to thread 24581
W20260321 08:57:31.793788 26178 debug-util.cc:398] Leaking SignalData structure 0x7b080008ef20 after lost signal to thread 26051
W20260321 08:57:31.794756 26178 debug-util.cc:398] Leaking SignalData structure 0x7b0800032ae0 after lost signal to thread 26181
W20260321 08:57:31.796121 26178 debug-util.cc:398] Leaking SignalData structure 0x7b080022dc80 after lost signal to thread 26308
W20260321 08:57:31.797188 26178 debug-util.cc:398] Leaking SignalData structure 0x7b08000f7fa0 after lost signal to thread 26386
W20260321 08:57:31.798238 26178 debug-util.cc:398] Leaking SignalData structure 0x7b08000a5800 after lost signal to thread 26464
W20260321 08:57:31.799904 26178 debug-util.cc:398] Leaking SignalData structure 0x7b08001b82e0 after lost signal to thread 26726
I20260321 08:57:31.804061 26282 raft_consensus.cc:1275] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 8 FOLLOWER]: Refusing update from remote peer d119b009c5744457b0555896e263fa35: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260321 08:57:31.805313 26857 consensus_queue.cc:1048] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:784: Failure
Expected equality of these values:
live_row_count
Which is: 67
table_info->GetMetrics()->live_row_count->value()
Which is: 134
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260321 08:57:43.481352 26870 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 784, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260321 08:57:43.488629 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:57:43.501959 26870 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8]
W20260321 08:57:43.508883 26326 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 6 similar messages]
W20260321 08:57:43.514091 26326 consensus_peers.cc:597] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 -> Peer de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365): Couldn't send request to peer de2947147ecb4d6eafa4aa039126f25b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:43.515079 26870 tablet_loader.cc:96] loaded metadata for tablet 3effe2075d034127a1ac31b507b8119a (table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:43.516768 26870 tablet_loader.cc:96] loaded metadata for tablet 51dc0b437c8c49aea324853455672c6c (table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8])
I20260321 08:57:43.521518 26870 tablet_loader.cc:96] loaded metadata for tablet b4eff2c92f994bb2a318dfe6097efacb (table NewTableName [id=aec8b423bf774e35beabf2b480fcfda8])
W20260321 08:57:43.522960 26326 consensus_peers.cc:597] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 -> Peer de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365): Couldn't send request to peer de2947147ecb4d6eafa4aa039126f25b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:43.523005 26870 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:43.531255 26870 catalog_manager.cc:1269] Loaded cluster ID: 9774d2365c48473a92ff616a29d5cfc9
I20260321 08:57:43.531535 26870 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:43.536942 26870 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:43.538425 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:43.539352 24578 tablet_replica.cc:333] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b: stopping tablet replica
I20260321 08:57:43.540189 24578 raft_consensus.cc:2243] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 8 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:43.541384 24578 raft_consensus.cc:2272] T 3effe2075d034127a1ac31b507b8119a P de2947147ecb4d6eafa4aa039126f25b [term 8 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.543337 26870 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8: Loaded TSK: 0
I20260321 08:57:43.544538 26870 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:43.545228 24578 tablet_replica.cc:333] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b: stopping tablet replica
I20260321 08:57:43.545943 24578 raft_consensus.cc:2243] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:43.546537 24578 raft_consensus.cc:2272] T 51dc0b437c8c49aea324853455672c6c P de2947147ecb4d6eafa4aa039126f25b [term 7 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.549034 24578 tablet_replica.cc:333] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b: stopping tablet replica
I20260321 08:57:43.549612 24578 raft_consensus.cc:2243] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 8 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:43.550139 24578 raft_consensus.cc:2272] T b4eff2c92f994bb2a318dfe6097efacb P de2947147ecb4d6eafa4aa039126f25b [term 8 FOLLOWER]: Raft consensus is shut down!
W20260321 08:57:43.550443 26326 consensus_peers.cc:597] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 -> Peer de2947147ecb4d6eafa4aa039126f25b (127.24.0.129:38365): Couldn't send request to peer de2947147ecb4d6eafa4aa039126f25b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:43.571987 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:57:43.586050 24578 tablet_server.cc:179] TabletServer@127.24.0.130:0 shutting down...
I20260321 08:57:43.602959 26387 heartbeater.cc:499] Master 127.24.0.188:45371 was elected leader, sending a full tablet report...
I20260321 08:57:43.618016 26145 catalog_manager.cc:5671] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 reported cstate change: term changed from 6 to 8, leader changed from de2947147ecb4d6eafa4aa039126f25b (127.24.0.129) to d119b009c5744457b0555896e263fa35 (127.24.0.130). New cstate: current_term: 8 leader_uuid: "d119b009c5744457b0555896e263fa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: HEALTHY } } }
I20260321 08:57:43.620079 26145 catalog_manager.cc:5671] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 reported cstate change: term changed from 7 to 8, leader changed from de2947147ecb4d6eafa4aa039126f25b (127.24.0.129) to d119b009c5744457b0555896e263fa35 (127.24.0.130). New cstate: current_term: 8 leader_uuid: "d119b009c5744457b0555896e263fa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2947147ecb4d6eafa4aa039126f25b" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d119b009c5744457b0555896e263fa35" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 41799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd3e8a46765d4e07b3d1087aca0be2be" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36993 } health_report { overall_health: HEALTHY } } }
I20260321 08:57:43.664088 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:43.664683 24578 tablet_replica.cc:333] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35: stopping tablet replica
I20260321 08:57:43.665167 24578 raft_consensus.cc:2243] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 8 LEADER]: Raft consensus shutting down.
I20260321 08:57:43.665903 24578 raft_consensus.cc:2272] T 3effe2075d034127a1ac31b507b8119a P d119b009c5744457b0555896e263fa35 [term 8 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.667605 24578 tablet_replica.cc:333] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35: stopping tablet replica
I20260321 08:57:43.668056 24578 raft_consensus.cc:2243] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 7 LEADER]: Raft consensus shutting down.
I20260321 08:57:43.668742 24578 raft_consensus.cc:2272] T 51dc0b437c8c49aea324853455672c6c P d119b009c5744457b0555896e263fa35 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.670329 24578 tablet_replica.cc:333] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35: stopping tablet replica
I20260321 08:57:43.670743 24578 raft_consensus.cc:2243] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 8 LEADER]: Raft consensus shutting down.
I20260321 08:57:43.671418 24578 raft_consensus.cc:2272] T b4eff2c92f994bb2a318dfe6097efacb P d119b009c5744457b0555896e263fa35 [term 8 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.693202 24578 tablet_server.cc:196] TabletServer@127.24.0.130:0 shutdown complete.
I20260321 08:57:43.707742 24578 tablet_server.cc:179] TabletServer@127.24.0.131:0 shutting down...
I20260321 08:57:43.734805 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:43.735380 24578 tablet_replica.cc:333] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be: stopping tablet replica
I20260321 08:57:43.735896 24578 raft_consensus.cc:2243] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 8 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:43.736419 24578 raft_consensus.cc:2272] T 3effe2075d034127a1ac31b507b8119a P dd3e8a46765d4e07b3d1087aca0be2be [term 8 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.738294 24578 tablet_replica.cc:333] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be: stopping tablet replica
I20260321 08:57:43.738802 24578 raft_consensus.cc:2243] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:43.739300 24578 raft_consensus.cc:2272] T 51dc0b437c8c49aea324853455672c6c P dd3e8a46765d4e07b3d1087aca0be2be [term 7 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.741050 24578 tablet_replica.cc:333] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be: stopping tablet replica
I20260321 08:57:43.741423 24578 raft_consensus.cc:2243] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 8 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:43.741777 24578 raft_consensus.cc:2272] T b4eff2c92f994bb2a318dfe6097efacb P dd3e8a46765d4e07b3d1087aca0be2be [term 8 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.791567 24578 tablet_server.cc:196] TabletServer@127.24.0.131:0 shutdown complete.
I20260321 08:57:43.804286 24578 master.cc:562] Master@127.24.0.190:45875 shutting down...
I20260321 08:57:43.827395 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 7 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:43.828001 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.828365 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 42e553cc2497492a9aa4dd5cb1d3ae74: stopping tablet replica
I20260321 08:57:43.868112 24578 master.cc:584] Master@127.24.0.190:45875 shutdown complete.
I20260321 08:57:43.882043 24578 master.cc:562] Master@127.24.0.189:38105 shutting down...
W20260321 08:57:43.892584 26130 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 -> Peer 42e553cc2497492a9aa4dd5cb1d3ae74 (127.24.0.190:45875): Couldn't send request to peer 42e553cc2497492a9aa4dd5cb1d3ae74. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.190:45875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:43.902662 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 7 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:43.903172 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 5c9e21016d934beab3beb647aa48006b [term 7 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.903487 24578 tablet_replica.cc:333] stopping tablet replica
W20260321 08:57:43.922814 26134 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 -> Peer 5c9e21016d934beab3beb647aa48006b (127.24.0.189:38105): Couldn't send request to peer 5c9e21016d934beab3beb647aa48006b. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.189:38105: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:43.923418 24578 master.cc:584] Master@127.24.0.189:38105 shutdown complete.
I20260321 08:57:43.937070 24578 master.cc:562] Master@127.24.0.188:45371 shutting down...
I20260321 08:57:43.952827 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 7 LEADER]: Raft consensus shutting down.
I20260321 08:57:43.953639 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:43.954030 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9af081622eaa4661a29feb3bf5d18cc8: stopping tablet replica
I20260321 08:57:43.994606 24578 master.cc:584] Master@127.24.0.188:45371 shutdown complete.
I20260321 08:57:44.018080 24578 test_util.cc:182] -----------------------------------------------
I20260321 08:57:44.018285 24578 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1774083407564530-24578-0
[ FAILED ] TsTabletManagerITest.TestTableStats (42298 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20260321 08:57:44.024183 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:35679
I20260321 08:57:44.025233 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:44.030702 26917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:44.030791 26918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:44.032127 26920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:44.033030 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:44.034646 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:44.034814 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:44.034953 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083464034937 us; error 0 us; skew 500 ppm
I20260321 08:57:44.035430 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:44.037786 24578 webserver.cc:492] Webserver started at http://127.24.0.190:44145/ using document root <none> and password file <none>
I20260321 08:57:44.038235 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:44.038393 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:44.038650 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:44.039734 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "b7aa23088e7c4a3292b304a62fb15b3f"
format_stamp: "Formatted at 2026-03-21 08:57:44 on dist-test-slave-p54g"
I20260321 08:57:44.043978 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260321 08:57:44.047031 26925 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:44.047696 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:57:44.047977 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "b7aa23088e7c4a3292b304a62fb15b3f"
format_stamp: "Formatted at 2026-03-21 08:57:44 on dist-test-slave-p54g"
I20260321 08:57:44.048226 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:44.075954 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:44.077142 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:44.113152 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:35679
I20260321 08:57:44.113242 26976 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:35679 every 8 connection(s)
I20260321 08:57:44.116720 26977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:44.126351 26977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f: Bootstrap starting.
I20260321 08:57:44.130769 26977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:44.134589 26977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f: No bootstrap required, opened a new log
I20260321 08:57:44.136476 26977 raft_consensus.cc:359] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7aa23088e7c4a3292b304a62fb15b3f" member_type: VOTER }
I20260321 08:57:44.136900 26977 raft_consensus.cc:385] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:44.137095 26977 raft_consensus.cc:740] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7aa23088e7c4a3292b304a62fb15b3f, State: Initialized, Role: FOLLOWER
I20260321 08:57:44.137585 26977 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7aa23088e7c4a3292b304a62fb15b3f" member_type: VOTER }
I20260321 08:57:44.138020 26977 raft_consensus.cc:399] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:44.138218 26977 raft_consensus.cc:493] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:44.138460 26977 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:44.142863 26977 raft_consensus.cc:515] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7aa23088e7c4a3292b304a62fb15b3f" member_type: VOTER }
I20260321 08:57:44.143321 26977 leader_election.cc:304] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7aa23088e7c4a3292b304a62fb15b3f; no voters:
I20260321 08:57:44.144340 26977 leader_election.cc:290] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:44.144706 26980 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:44.145876 26980 raft_consensus.cc:697] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 1 LEADER]: Becoming Leader. State: Replica: b7aa23088e7c4a3292b304a62fb15b3f, State: Running, Role: LEADER
I20260321 08:57:44.146560 26980 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7aa23088e7c4a3292b304a62fb15b3f" member_type: VOTER }
I20260321 08:57:44.147069 26977 sys_catalog.cc:565] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:44.149324 26982 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7aa23088e7c4a3292b304a62fb15b3f. Latest consensus state: current_term: 1 leader_uuid: "b7aa23088e7c4a3292b304a62fb15b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7aa23088e7c4a3292b304a62fb15b3f" member_type: VOTER } }
I20260321 08:57:44.149353 26981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7aa23088e7c4a3292b304a62fb15b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7aa23088e7c4a3292b304a62fb15b3f" member_type: VOTER } }
I20260321 08:57:44.150012 26982 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:44.150288 26981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:44.154129 26985 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:44.158224 26985 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:44.160925 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:57:44.165879 26985 catalog_manager.cc:1357] Generated new cluster ID: 3a3c6ae46e84425186e11dd6c41b1e39
I20260321 08:57:44.166141 26985 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:44.177459 26985 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:57:44.178596 26985 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:44.192544 26985 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f: Generated new TSK 0
I20260321 08:57:44.193095 26985 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:44.227280 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:44.232964 26998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:44.234018 26999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:44.235380 24578 server_base.cc:1061] running on GCE node
W20260321 08:57:44.235291 27001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:44.236649 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:44.236843 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:44.237063 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083464237042 us; error 0 us; skew 500 ppm
I20260321 08:57:44.237579 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:44.239845 24578 webserver.cc:492] Webserver started at http://127.24.0.129:40285/ using document root <none> and password file <none>
I20260321 08:57:44.240370 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:44.240537 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:44.240768 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:44.241803 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "747d286ce46948cdb42e7fb0f4500a3d"
format_stamp: "Formatted at 2026-03-21 08:57:44 on dist-test-slave-p54g"
I20260321 08:57:44.246088 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260321 08:57:44.249260 27006 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:44.250063 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260321 08:57:44.250337 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "747d286ce46948cdb42e7fb0f4500a3d"
format_stamp: "Formatted at 2026-03-21 08:57:44 on dist-test-slave-p54g"
I20260321 08:57:44.250613 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:44.271404 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:44.272490 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:44.273869 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:44.276021 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:44.276206 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:44.276417 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:44.276578 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:44.314019 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:33189
I20260321 08:57:44.314118 27068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:33189 every 8 connection(s)
I20260321 08:57:44.319041 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:44.326318 27073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:44.327190 27074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:44.341568 27069 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35679
W20260321 08:57:44.341765 27076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:44.342020 27069 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:44.341800 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:44.342952 27069 heartbeater.cc:507] Master 127.24.0.190:35679 requested a full tablet report, sending...
I20260321 08:57:44.343056 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:44.343302 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:44.343456 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083464343437 us; error 0 us; skew 500 ppm
I20260321 08:57:44.344128 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:44.344942 26942 ts_manager.cc:194] Registered new tserver with Master: 747d286ce46948cdb42e7fb0f4500a3d (127.24.0.129:33189)
I20260321 08:57:44.346662 24578 webserver.cc:492] Webserver started at http://127.24.0.130:35523/ using document root <none> and password file <none>
I20260321 08:57:44.347297 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:44.347352 26942 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42006
I20260321 08:57:44.347627 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:44.348027 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:44.349260 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-1-root/instance:
uuid: "c0ee332ca2254eafba4f2723358ad9a4"
format_stamp: "Formatted at 2026-03-21 08:57:44 on dist-test-slave-p54g"
I20260321 08:57:44.353302 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260321 08:57:44.356498 27081 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:44.357164 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260321 08:57:44.357424 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-1-root
uuid: "c0ee332ca2254eafba4f2723358ad9a4"
format_stamp: "Formatted at 2026-03-21 08:57:44 on dist-test-slave-p54g"
I20260321 08:57:44.357672 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:44.368446 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:44.369535 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:44.370846 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:44.372881 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:44.373066 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:44.373276 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:44.373428 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:44.410032 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.130:38671
I20260321 08:57:44.410115 27143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.130:38671 every 8 connection(s)
I20260321 08:57:44.414462 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:44.421784 27147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:44.422431 27148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:44.425385 27144 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35679
W20260321 08:57:44.425475 27150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:44.425760 27144 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:44.426486 27144 heartbeater.cc:507] Master 127.24.0.190:35679 requested a full tablet report, sending...
I20260321 08:57:44.428272 26942 ts_manager.cc:194] Registered new tserver with Master: c0ee332ca2254eafba4f2723358ad9a4 (127.24.0.130:38671)
I20260321 08:57:44.428431 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:44.429262 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:44.429457 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:44.429622 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083464429607 us; error 0 us; skew 500 ppm
I20260321 08:57:44.429701 26942 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42014
I20260321 08:57:44.430153 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:44.432678 24578 webserver.cc:492] Webserver started at http://127.24.0.131:41267/ using document root <none> and password file <none>
I20260321 08:57:44.433125 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:44.433269 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:44.433460 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:44.434485 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-2-root/instance:
uuid: "25b6e8a7438449eea9ca7d917bdb0947"
format_stamp: "Formatted at 2026-03-21 08:57:44 on dist-test-slave-p54g"
I20260321 08:57:44.438601 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260321 08:57:44.441635 27155 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:44.442405 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260321 08:57:44.442763 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-2-root
uuid: "25b6e8a7438449eea9ca7d917bdb0947"
format_stamp: "Formatted at 2026-03-21 08:57:44 on dist-test-slave-p54g"
I20260321 08:57:44.443007 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:44.453474 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:44.454566 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:44.455868 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:44.458179 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:44.458353 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:44.458545 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:44.458667 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:44.495394 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.131:37641
I20260321 08:57:44.495512 27217 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.131:37641 every 8 connection(s)
I20260321 08:57:44.507797 27218 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35679
I20260321 08:57:44.508152 27218 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:44.508817 27218 heartbeater.cc:507] Master 127.24.0.190:35679 requested a full tablet report, sending...
I20260321 08:57:44.510491 26942 ts_manager.cc:194] Registered new tserver with Master: 25b6e8a7438449eea9ca7d917bdb0947 (127.24.0.131:37641)
I20260321 08:57:44.510627 24578 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012375411s
I20260321 08:57:44.511754 26942 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42026
I20260321 08:57:44.541529 26942 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260321 08:57:44.542997 26942 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260321 08:57:44.583117 27109 tablet_service.cc:1511] Processing CreateTablet for tablet e61fcd666acf467192a33e63f56f242c (DEFAULT_TABLE table=test-table [id=28c1eb8ebaf34175b0d9c13c9c1345e0]), partition=
I20260321 08:57:44.584331 27109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e61fcd666acf467192a33e63f56f242c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:44.585399 27034 tablet_service.cc:1511] Processing CreateTablet for tablet e61fcd666acf467192a33e63f56f242c (DEFAULT_TABLE table=test-table [id=28c1eb8ebaf34175b0d9c13c9c1345e0]), partition=
I20260321 08:57:44.586423 27034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e61fcd666acf467192a33e63f56f242c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:44.592890 27183 tablet_service.cc:1511] Processing CreateTablet for tablet e61fcd666acf467192a33e63f56f242c (DEFAULT_TABLE table=test-table [id=28c1eb8ebaf34175b0d9c13c9c1345e0]), partition=
I20260321 08:57:44.593761 27183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e61fcd666acf467192a33e63f56f242c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:44.599937 27234 tablet_bootstrap.cc:492] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: Bootstrap starting.
I20260321 08:57:44.601763 27235 tablet_bootstrap.cc:492] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Bootstrap starting.
I20260321 08:57:44.605484 27234 tablet_bootstrap.cc:654] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:44.607667 27236 tablet_bootstrap.cc:492] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: Bootstrap starting.
I20260321 08:57:44.608027 27235 tablet_bootstrap.cc:654] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:44.610354 27234 tablet_bootstrap.cc:492] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: No bootstrap required, opened a new log
I20260321 08:57:44.610771 27234 ts_tablet_manager.cc:1403] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260321 08:57:44.612668 27235 tablet_bootstrap.cc:492] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: No bootstrap required, opened a new log
I20260321 08:57:44.613021 27235 ts_tablet_manager.cc:1403] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260321 08:57:44.613500 27234 raft_consensus.cc:359] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.614248 27234 raft_consensus.cc:385] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:44.614565 27234 raft_consensus.cc:740] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ee332ca2254eafba4f2723358ad9a4, State: Initialized, Role: FOLLOWER
I20260321 08:57:44.615458 27236 tablet_bootstrap.cc:654] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:44.615239 27235 raft_consensus.cc:359] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.615213 27234 consensus_queue.cc:260] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.616035 27235 raft_consensus.cc:385] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:44.616338 27235 raft_consensus.cc:740] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747d286ce46948cdb42e7fb0f4500a3d, State: Initialized, Role: FOLLOWER
I20260321 08:57:44.616868 27235 consensus_queue.cc:260] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.618805 27069 heartbeater.cc:499] Master 127.24.0.190:35679 was elected leader, sending a full tablet report...
I20260321 08:57:44.620538 27235 ts_tablet_manager.cc:1434] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260321 08:57:44.620883 27235 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260321 08:57:44.625609 27144 heartbeater.cc:499] Master 127.24.0.190:35679 was elected leader, sending a full tablet report...
I20260321 08:57:44.625849 27236 tablet_bootstrap.cc:492] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: No bootstrap required, opened a new log
I20260321 08:57:44.626236 27236 ts_tablet_manager.cc:1403] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.006s
I20260321 08:57:44.626827 27234 ts_tablet_manager.cc:1434] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: Time spent starting tablet: real 0.016s user 0.007s sys 0.004s
W20260321 08:57:44.627053 27234 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260321 08:57:44.628331 27236 raft_consensus.cc:359] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.628821 27236 raft_consensus.cc:385] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:44.629024 27236 raft_consensus.cc:740] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b6e8a7438449eea9ca7d917bdb0947, State: Initialized, Role: FOLLOWER
I20260321 08:57:44.629541 27236 consensus_queue.cc:260] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.631316 27218 heartbeater.cc:499] Master 127.24.0.190:35679 was elected leader, sending a full tablet report...
I20260321 08:57:44.631841 27236 ts_tablet_manager.cc:1434] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260321 08:57:44.632130 27236 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
W20260321 08:57:44.663586 27145 tablet.cc:2379] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260321 08:57:44.724097 27242 raft_consensus.cc:493] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:44.724575 27242 raft_consensus.cc:515] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.726385 27242 leader_election.cc:290] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ee332ca2254eafba4f2723358ad9a4 (127.24.0.130:38671), 747d286ce46948cdb42e7fb0f4500a3d (127.24.0.129:33189)
I20260321 08:57:44.735216 27119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61fcd666acf467192a33e63f56f242c" candidate_uuid: "25b6e8a7438449eea9ca7d917bdb0947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ee332ca2254eafba4f2723358ad9a4" is_pre_election: true
I20260321 08:57:44.735852 27119 raft_consensus.cc:2468] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25b6e8a7438449eea9ca7d917bdb0947 in term 0.
I20260321 08:57:44.736461 27044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61fcd666acf467192a33e63f56f242c" candidate_uuid: "25b6e8a7438449eea9ca7d917bdb0947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747d286ce46948cdb42e7fb0f4500a3d" is_pre_election: true
I20260321 08:57:44.737076 27044 raft_consensus.cc:2468] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25b6e8a7438449eea9ca7d917bdb0947 in term 0.
I20260321 08:57:44.736984 27156 leader_election.cc:304] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25b6e8a7438449eea9ca7d917bdb0947, c0ee332ca2254eafba4f2723358ad9a4; no voters:
I20260321 08:57:44.737914 27242 raft_consensus.cc:2804] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:57:44.738303 27242 raft_consensus.cc:493] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:44.738543 27242 raft_consensus.cc:3060] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:44.743220 27242 raft_consensus.cc:515] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.744593 27242 leader_election.cc:290] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [CANDIDATE]: Term 1 election: Requested vote from peers c0ee332ca2254eafba4f2723358ad9a4 (127.24.0.130:38671), 747d286ce46948cdb42e7fb0f4500a3d (127.24.0.129:33189)
I20260321 08:57:44.745328 27119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61fcd666acf467192a33e63f56f242c" candidate_uuid: "25b6e8a7438449eea9ca7d917bdb0947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ee332ca2254eafba4f2723358ad9a4"
I20260321 08:57:44.745513 27044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61fcd666acf467192a33e63f56f242c" candidate_uuid: "25b6e8a7438449eea9ca7d917bdb0947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747d286ce46948cdb42e7fb0f4500a3d"
I20260321 08:57:44.745762 27119 raft_consensus.cc:3060] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:44.746002 27044 raft_consensus.cc:3060] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:44.751142 27119 raft_consensus.cc:2468] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25b6e8a7438449eea9ca7d917bdb0947 in term 1.
I20260321 08:57:44.751448 27044 raft_consensus.cc:2468] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25b6e8a7438449eea9ca7d917bdb0947 in term 1.
I20260321 08:57:44.752080 27156 leader_election.cc:304] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25b6e8a7438449eea9ca7d917bdb0947, c0ee332ca2254eafba4f2723358ad9a4; no voters:
I20260321 08:57:44.752689 27242 raft_consensus.cc:2804] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:44.753450 27242 raft_consensus.cc:697] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 1 LEADER]: Becoming Leader. State: Replica: 25b6e8a7438449eea9ca7d917bdb0947, State: Running, Role: LEADER
I20260321 08:57:44.754175 27242 consensus_queue.cc:237] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.761611 26941 catalog_manager.cc:5671] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25b6e8a7438449eea9ca7d917bdb0947 (127.24.0.131). New cstate: current_term: 1 leader_uuid: "25b6e8a7438449eea9ca7d917bdb0947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } health_report { overall_health: HEALTHY } } }
I20260321 08:57:44.810995 27034 tablet_service.cc:1558] Processing DeleteTablet for tablet e61fcd666acf467192a33e63f56f242c with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43944
I20260321 08:57:44.812529 27247 tablet_replica.cc:333] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: stopping tablet replica
I20260321 08:57:44.813063 27247 raft_consensus.cc:2243] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:44.813493 27247 raft_consensus.cc:2272] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:44.815414 27247 ts_tablet_manager.cc:1916] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 08:57:44.825419 27247 ts_tablet_manager.cc:1929] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20260321 08:57:44.825736 27247 log.cc:1199] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Deleting WAL directory at /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-0-root/wals/e61fcd666acf467192a33e63f56f242c
I20260321 08:57:44.829540 27248 ts_tablet_manager.cc:933] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Initiating tablet copy from peer 25b6e8a7438449eea9ca7d917bdb0947 (127.24.0.131:37641)
I20260321 08:57:44.831166 27248 tablet_copy_client.cc:323] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: tablet copy: Beginning tablet copy session from remote peer at address 127.24.0.131:37641
I20260321 08:57:44.839111 27203 tablet_copy_service.cc:140] P 25b6e8a7438449eea9ca7d917bdb0947: Received BeginTabletCopySession request for tablet e61fcd666acf467192a33e63f56f242c from peer 747d286ce46948cdb42e7fb0f4500a3d ({username='slave'} at 127.0.0.1:48278)
I20260321 08:57:44.839589 27203 tablet_copy_service.cc:161] P 25b6e8a7438449eea9ca7d917bdb0947: Beginning new tablet copy session on tablet e61fcd666acf467192a33e63f56f242c from peer 747d286ce46948cdb42e7fb0f4500a3d at {username='slave'} at 127.0.0.1:48278: session id = 747d286ce46948cdb42e7fb0f4500a3d-e61fcd666acf467192a33e63f56f242c
I20260321 08:57:44.844527 27203 tablet_copy_source_session.cc:215] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 08:57:44.846864 27248 ts_tablet_manager.cc:1916] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260321 08:57:44.856468 27248 ts_tablet_manager.cc:1929] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20260321 08:57:44.857031 27248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e61fcd666acf467192a33e63f56f242c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:44.862035 27248 tablet_copy_client.cc:806] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: tablet copy: Starting download of 0 data blocks...
I20260321 08:57:44.862524 27248 tablet_copy_client.cc:670] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: tablet copy: Starting download of 1 WAL segments...
I20260321 08:57:44.864104 26941 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42032:
table { table_id: "28c1eb8ebaf34175b0d9c13c9c1345e0" table_name: "test-table" }
I20260321 08:57:44.864661 26941 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42032:
table { table_id: "28c1eb8ebaf34175b0d9c13c9c1345e0" table_name: "test-table" }
I20260321 08:57:44.866149 27248 tablet_copy_client.cc:538] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 08:57:44.872581 27248 tablet_bootstrap.cc:492] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Bootstrap starting.
I20260321 08:57:44.874367 26941 catalog_manager.cc:5958] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f: Sending DeleteTablet for 3 replicas of tablet e61fcd666acf467192a33e63f56f242c
I20260321 08:57:44.876204 27109 tablet_service.cc:1558] Processing DeleteTablet for tablet e61fcd666acf467192a33e63f56f242c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-21 08:57:44 UTC) from {username='slave'} at 127.0.0.1:36994
I20260321 08:57:44.876857 27183 tablet_service.cc:1558] Processing DeleteTablet for tablet e61fcd666acf467192a33e63f56f242c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-21 08:57:44 UTC) from {username='slave'} at 127.0.0.1:48248
I20260321 08:57:44.878175 27034 tablet_service.cc:1558] Processing DeleteTablet for tablet e61fcd666acf467192a33e63f56f242c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-21 08:57:44 UTC) from {username='slave'} at 127.0.0.1:43922
W20260321 08:57:44.880208 26926 catalog_manager.cc:4982] TS 747d286ce46948cdb42e7fb0f4500a3d (127.24.0.129:33189): delete failed for tablet e61fcd666acf467192a33e63f56f242c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e61fcd666acf467192a33e63f56f242c already in progress: copying tablet
I20260321 08:57:44.881693 27250 tablet_replica.cc:333] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: stopping tablet replica
I20260321 08:57:44.882095 27251 tablet_replica.cc:333] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: stopping tablet replica
I20260321 08:57:44.882438 27250 raft_consensus.cc:2243] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:44.882797 27251 raft_consensus.cc:2243] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:44.882988 27250 raft_consensus.cc:2272] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:44.883586 27251 pending_rounds.cc:70] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: Trying to abort 1 pending ops.
I20260321 08:57:44.883905 27251 pending_rounds.cc:77] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7266645871639576576 op_type: NO_OP noop_request { }
I20260321 08:57:44.884370 27251 raft_consensus.cc:2889] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260321 08:57:44.884727 27251 raft_consensus.cc:2272] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:44.885754 27250 ts_tablet_manager.cc:1916] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260321 08:57:44.887329 27251 ts_tablet_manager.cc:1916] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260321 08:57:44.887959 27248 tablet_bootstrap.cc:492] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260321 08:57:44.888695 27248 tablet_bootstrap.cc:492] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Bootstrap complete.
I20260321 08:57:44.889178 27248 ts_tablet_manager.cc:1403] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260321 08:57:44.891170 27248 raft_consensus.cc:359] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.891796 27248 raft_consensus.cc:740] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747d286ce46948cdb42e7fb0f4500a3d, State: Initialized, Role: FOLLOWER
I20260321 08:57:44.892410 27248 consensus_queue.cc:260] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ee332ca2254eafba4f2723358ad9a4" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 38671 } } peers { permanent_uuid: "747d286ce46948cdb42e7fb0f4500a3d" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33189 } } peers { permanent_uuid: "25b6e8a7438449eea9ca7d917bdb0947" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 37641 } }
I20260321 08:57:44.894219 27248 ts_tablet_manager.cc:1434] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260321 08:57:44.894488 27248 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260321 08:57:44.898382 27034 tablet_service.cc:1558] Processing DeleteTablet for tablet e61fcd666acf467192a33e63f56f242c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-21 08:57:44 UTC) from {username='slave'} at 127.0.0.1:43922
W20260321 08:57:44.899600 26926 catalog_manager.cc:4982] TS 747d286ce46948cdb42e7fb0f4500a3d (127.24.0.129:33189): delete failed for tablet e61fcd666acf467192a33e63f56f242c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e61fcd666acf467192a33e63f56f242c already in progress: copying tablet
I20260321 08:57:44.900229 27250 ts_tablet_manager.cc:1929] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260321 08:57:44.900640 27250 log.cc:1199] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: Deleting WAL directory at /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-1-root/wals/e61fcd666acf467192a33e63f56f242c
I20260321 08:57:44.901279 27250 ts_tablet_manager.cc:1950] T e61fcd666acf467192a33e63f56f242c P c0ee332ca2254eafba4f2723358ad9a4: Deleting consensus metadata
I20260321 08:57:44.901319 27251 ts_tablet_manager.cc:1929] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260321 08:57:44.901791 27251 log.cc:1199] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: Deleting WAL directory at /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-2-root/wals/e61fcd666acf467192a33e63f56f242c
I20260321 08:57:44.902568 27251 ts_tablet_manager.cc:1950] T e61fcd666acf467192a33e63f56f242c P 25b6e8a7438449eea9ca7d917bdb0947: Deleting consensus metadata
I20260321 08:57:44.903760 26926 catalog_manager.cc:5002] TS 25b6e8a7438449eea9ca7d917bdb0947 (127.24.0.131:37641): tablet e61fcd666acf467192a33e63f56f242c (table test-table [id=28c1eb8ebaf34175b0d9c13c9c1345e0]) successfully deleted
I20260321 08:57:44.904371 26926 catalog_manager.cc:5002] TS c0ee332ca2254eafba4f2723358ad9a4 (127.24.0.130:38671): tablet e61fcd666acf467192a33e63f56f242c (table test-table [id=28c1eb8ebaf34175b0d9c13c9c1345e0]) successfully deleted
I20260321 08:57:44.946694 27203 tablet_copy_service.cc:342] P 25b6e8a7438449eea9ca7d917bdb0947: Request end of tablet copy session 747d286ce46948cdb42e7fb0f4500a3d-e61fcd666acf467192a33e63f56f242c received from {username='slave'} at 127.0.0.1:48278
I20260321 08:57:44.947117 27203 tablet_copy_service.cc:434] P 25b6e8a7438449eea9ca7d917bdb0947: ending tablet copy session 747d286ce46948cdb42e7fb0f4500a3d-e61fcd666acf467192a33e63f56f242c on tablet e61fcd666acf467192a33e63f56f242c with peer 747d286ce46948cdb42e7fb0f4500a3d
I20260321 08:57:44.950687 27034 tablet_service.cc:1558] Processing DeleteTablet for tablet e61fcd666acf467192a33e63f56f242c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-21 08:57:44 UTC) from {username='slave'} at 127.0.0.1:43922
I20260321 08:57:44.951486 27247 tablet_replica.cc:333] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: stopping tablet replica
I20260321 08:57:44.952270 27247 raft_consensus.cc:2243] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:44.952703 27247 pending_rounds.cc:70] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Trying to abort 1 pending ops.
I20260321 08:57:44.952953 27247 pending_rounds.cc:77] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7266645871639576576 op_type: NO_OP noop_request { }
I20260321 08:57:44.953308 27247 raft_consensus.cc:2889] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260321 08:57:44.953665 27247 raft_consensus.cc:2272] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:44.955704 27247 ts_tablet_manager.cc:1916] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260321 08:57:44.965035 27247 ts_tablet_manager.cc:1929] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260321 08:57:44.965289 27247 log.cc:1199] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Deleting WAL directory at /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1774083407564530-24578-0/minicluster-data/ts-0-root/wals/e61fcd666acf467192a33e63f56f242c
I20260321 08:57:44.965790 27247 ts_tablet_manager.cc:1950] T e61fcd666acf467192a33e63f56f242c P 747d286ce46948cdb42e7fb0f4500a3d: Deleting consensus metadata
I20260321 08:57:44.967782 26926 catalog_manager.cc:5002] TS 747d286ce46948cdb42e7fb0f4500a3d (127.24.0.129:33189): tablet e61fcd666acf467192a33e63f56f242c (table test-table [id=28c1eb8ebaf34175b0d9c13c9c1345e0]) successfully deleted
I20260321 08:57:45.011933 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:57:45.031841 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:45.048678 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:57:45.057228 24578 tablet_server.cc:179] TabletServer@127.24.0.130:0 shutting down...
I20260321 08:57:45.071552 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:45.087284 24578 tablet_server.cc:196] TabletServer@127.24.0.130:0 shutdown complete.
I20260321 08:57:45.095623 24578 tablet_server.cc:179] TabletServer@127.24.0.131:0 shutting down...
I20260321 08:57:45.110948 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:45.127364 24578 tablet_server.cc:196] TabletServer@127.24.0.131:0 shutdown complete.
I20260321 08:57:45.135607 24578 master.cc:562] Master@127.24.0.190:35679 shutting down...
I20260321 08:57:45.152617 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:45.153111 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:45.153401 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P b7aa23088e7c4a3292b304a62fb15b3f: stopping tablet replica
I20260321 08:57:45.170245 24578 master.cc:584] Master@127.24.0.190:35679 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1169 ms)
[----------] 4 tests from TsTabletManagerITest (46471 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20260321 08:57:45.193997 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:35585
I20260321 08:57:45.194998 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:45.199417 27257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:45.200346 27258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:45.201526 27260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:45.201670 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:45.202603 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:45.202770 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:45.202884 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083465202873 us; error 0 us; skew 500 ppm
I20260321 08:57:45.203315 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:45.205402 24578 webserver.cc:492] Webserver started at http://127.24.0.190:36845/ using document root <none> and password file <none>
I20260321 08:57:45.205817 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:45.205958 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:45.206159 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:45.207125 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "54af501bc7ed4f3aa0760f627eb93acf"
format_stamp: "Formatted at 2026-03-21 08:57:45 on dist-test-slave-p54g"
I20260321 08:57:45.211239 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260321 08:57:45.214345 27265 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:45.215001 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260321 08:57:45.215245 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "54af501bc7ed4f3aa0760f627eb93acf"
format_stamp: "Formatted at 2026-03-21 08:57:45 on dist-test-slave-p54g"
I20260321 08:57:45.215508 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:45.226447 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:45.227375 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:45.260236 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:35585
I20260321 08:57:45.260293 27316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:35585 every 8 connection(s)
I20260321 08:57:45.263895 27317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:45.273227 27317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf: Bootstrap starting.
I20260321 08:57:45.277177 27317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:45.280650 27317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf: No bootstrap required, opened a new log
I20260321 08:57:45.282403 27317 raft_consensus.cc:359] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54af501bc7ed4f3aa0760f627eb93acf" member_type: VOTER }
I20260321 08:57:45.282739 27317 raft_consensus.cc:385] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:45.282912 27317 raft_consensus.cc:740] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54af501bc7ed4f3aa0760f627eb93acf, State: Initialized, Role: FOLLOWER
I20260321 08:57:45.283396 27317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54af501bc7ed4f3aa0760f627eb93acf" member_type: VOTER }
I20260321 08:57:45.283804 27317 raft_consensus.cc:399] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:45.284004 27317 raft_consensus.cc:493] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:45.284202 27317 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:45.288012 27317 raft_consensus.cc:515] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54af501bc7ed4f3aa0760f627eb93acf" member_type: VOTER }
I20260321 08:57:45.288436 27317 leader_election.cc:304] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54af501bc7ed4f3aa0760f627eb93acf; no voters:
I20260321 08:57:45.289387 27317 leader_election.cc:290] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:45.289657 27320 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:45.290781 27320 raft_consensus.cc:697] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 1 LEADER]: Becoming Leader. State: Replica: 54af501bc7ed4f3aa0760f627eb93acf, State: Running, Role: LEADER
I20260321 08:57:45.291417 27320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54af501bc7ed4f3aa0760f627eb93acf" member_type: VOTER }
I20260321 08:57:45.292074 27317 sys_catalog.cc:565] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:45.296131 27322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54af501bc7ed4f3aa0760f627eb93acf. Latest consensus state: current_term: 1 leader_uuid: "54af501bc7ed4f3aa0760f627eb93acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54af501bc7ed4f3aa0760f627eb93acf" member_type: VOTER } }
I20260321 08:57:45.296677 27322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:45.297863 27321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54af501bc7ed4f3aa0760f627eb93acf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54af501bc7ed4f3aa0760f627eb93acf" member_type: VOTER } }
I20260321 08:57:45.298118 27328 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:45.298491 27321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:45.303726 27328 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:45.304734 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:57:45.311487 27328 catalog_manager.cc:1357] Generated new cluster ID: c3d5b607dc3d4695a434d64f66626a63
I20260321 08:57:45.311775 27328 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:45.328816 27328 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:57:45.329950 27328 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:45.344183 27328 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf: Generated new TSK 0
I20260321 08:57:45.344750 27328 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:45.370735 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:45.376122 27338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:45.376989 27339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:45.378780 27341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:45.378834 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:45.379712 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:45.379920 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:45.380061 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083465380044 us; error 0 us; skew 500 ppm
I20260321 08:57:45.380534 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:45.382611 24578 webserver.cc:492] Webserver started at http://127.24.0.129:33817/ using document root <none> and password file <none>
I20260321 08:57:45.383069 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:45.383235 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:45.383483 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:45.384510 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "5fc018b63db446139e033902fc0371ef"
format_stamp: "Formatted at 2026-03-21 08:57:45 on dist-test-slave-p54g"
I20260321 08:57:45.388641 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260321 08:57:45.391628 27346 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:45.392371 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260321 08:57:45.392630 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "5fc018b63db446139e033902fc0371ef"
format_stamp: "Formatted at 2026-03-21 08:57:45 on dist-test-slave-p54g"
I20260321 08:57:45.392905 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:45.405036 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:45.406056 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:45.410701 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:45.411010 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:45.411300 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:45.411516 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:45.456211 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:44999
I20260321 08:57:45.456333 27416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:44999 every 8 connection(s)
I20260321 08:57:45.460359 27417 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35585
I20260321 08:57:45.460726 27417 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:45.461360 27417 heartbeater.cc:507] Master 127.24.0.190:35585 requested a full tablet report, sending...
I20260321 08:57:45.463104 27282 ts_manager.cc:194] Registered new tserver with Master: 5fc018b63db446139e033902fc0371ef (127.24.0.129:44999)
I20260321 08:57:45.463935 24578 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004845278s
I20260321 08:57:45.464730 27282 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47304
I20260321 08:57:45.492689 27382 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260321 08:57:45.493871 27382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:45.504809 27427 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: Bootstrap starting.
I20260321 08:57:45.508955 27427 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:45.512625 27427 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: No bootstrap required, opened a new log
I20260321 08:57:45.512959 27427 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260321 08:57:45.514724 27427 raft_consensus.cc:359] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } }
I20260321 08:57:45.515094 27427 raft_consensus.cc:385] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:45.515298 27427 raft_consensus.cc:740] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc018b63db446139e033902fc0371ef, State: Initialized, Role: FOLLOWER
I20260321 08:57:45.515797 27427 consensus_queue.cc:260] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } }
I20260321 08:57:45.516299 27427 raft_consensus.cc:399] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:45.516506 27427 raft_consensus.cc:493] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:45.516782 27427 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:45.521160 27427 raft_consensus.cc:515] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } }
I20260321 08:57:45.521678 27427 leader_election.cc:304] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc018b63db446139e033902fc0371ef; no voters:
I20260321 08:57:45.522822 27427 leader_election.cc:290] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:45.523133 27429 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:45.524541 27429 raft_consensus.cc:697] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 1 LEADER]: Becoming Leader. State: Replica: 5fc018b63db446139e033902fc0371ef, State: Running, Role: LEADER
I20260321 08:57:45.524549 27417 heartbeater.cc:499] Master 127.24.0.190:35585 was elected leader, sending a full tablet report...
I20260321 08:57:45.525272 27427 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260321 08:57:45.525317 27429 consensus_queue.cc:237] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } }
I20260321 08:57:45.529351 27431 tablet_replica.cc:442] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: TxnStatusTablet state changed. Reason: New leader 5fc018b63db446139e033902fc0371ef. Latest consensus state: current_term: 1 leader_uuid: "5fc018b63db446139e033902fc0371ef" committed_config { opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } } }
I20260321 08:57:45.529587 27430 tablet_replica.cc:442] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fc018b63db446139e033902fc0371ef" committed_config { opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } } }
I20260321 08:57:45.529937 27431 tablet_replica.cc:445] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: This TxnStatusTablet replica's current role is: LEADER
I20260321 08:57:45.530241 27430 tablet_replica.cc:445] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: This TxnStatusTablet replica's current role is: LEADER
I20260321 08:57:45.532387 27434 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
W20260321 08:57:45.532483 27282 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:45.532806 27434 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260321 08:57:45.537089 27282 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:45.570055 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:57:45.588521 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:45.589252 24578 tablet_replica.cc:333] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: stopping tablet replica
I20260321 08:57:45.589768 24578 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:45.590210 24578 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:45.600647 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:57:45.612466 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:45.617827 27437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:45.619408 27438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:45.621807 27440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:45.622550 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:45.623399 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:45.623581 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:45.623729 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083465623712 us; error 0 us; skew 500 ppm
I20260321 08:57:45.624222 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:45.626765 24578 webserver.cc:492] Webserver started at http://127.24.0.129:33817/ using document root <none> and password file <none>
I20260321 08:57:45.627202 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:45.627352 24578 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260321 08:57:45.630750 24578 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260321 08:57:45.633422 27445 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:45.634218 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260321 08:57:45.634472 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "5fc018b63db446139e033902fc0371ef"
format_stamp: "Formatted at 2026-03-21 08:57:45 on dist-test-slave-p54g"
I20260321 08:57:45.634734 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:45.650437 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:45.651363 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:45.658329 27454 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20260321 08:57:45.665839 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260321 08:57:45.666126 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.001s
I20260321 08:57:45.666420 24578 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260321 08:57:45.672497 27454 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: Bootstrap starting.
I20260321 08:57:45.675969 24578 ts_tablet_manager.cc:616] Registered 1 tablets
I20260321 08:57:45.676237 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.010s user 0.003s sys 0.003s
I20260321 08:57:45.706344 27454 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 08:57:45.707306 27454 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: Bootstrap complete.
I20260321 08:57:45.708014 27454 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.013s
I20260321 08:57:45.710273 27454 raft_consensus.cc:359] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } }
I20260321 08:57:45.710837 27454 raft_consensus.cc:740] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc018b63db446139e033902fc0371ef, State: Initialized, Role: FOLLOWER
I20260321 08:57:45.711473 27454 consensus_queue.cc:260] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } }
I20260321 08:57:45.711961 27454 raft_consensus.cc:399] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:45.712224 27454 raft_consensus.cc:493] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:45.712476 27454 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:45.717397 27454 raft_consensus.cc:515] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } }
I20260321 08:57:45.718065 27454 leader_election.cc:304] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc018b63db446139e033902fc0371ef; no voters:
I20260321 08:57:45.719455 27454 leader_election.cc:290] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260321 08:57:45.720387 27468 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 2 FOLLOWER]: Leader election won for term 2
I20260321 08:57:45.728248 27454 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef: Time spent starting tablet: real 0.020s user 0.011s sys 0.009s
I20260321 08:57:45.729781 27468 raft_consensus.cc:697] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 2 LEADER]: Becoming Leader. State: Replica: 5fc018b63db446139e033902fc0371ef, State: Running, Role: LEADER
I20260321 08:57:45.730564 27468 consensus_queue.cc:237] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } }
I20260321 08:57:45.740667 27472 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 5fc018b63db446139e033902fc0371ef. Latest consensus state: current_term: 2 leader_uuid: "5fc018b63db446139e033902fc0371ef" committed_config { opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } } }
I20260321 08:57:45.741264 27472 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260321 08:57:45.752570 27470 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5fc018b63db446139e033902fc0371ef" committed_config { opid_index: -1 peers { permanent_uuid: "5fc018b63db446139e033902fc0371ef" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 44999 } } }
I20260321 08:57:45.753157 27470 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260321 08:57:45.754349 27480 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260321 08:57:45.754822 27480 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260321 08:57:45.784972 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:44999
I20260321 08:57:45.785034 27523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:44999 every 8 connection(s)
I20260321 08:57:45.791049 27524 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35585
I20260321 08:57:45.791448 27524 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:45.792418 27524 heartbeater.cc:507] Master 127.24.0.190:35585 requested a full tablet report, sending...
I20260321 08:57:45.792414 24578 tablet_server.cc:179] TabletServer@127.24.0.129:44999 shutting down...
I20260321 08:57:45.795917 27282 ts_manager.cc:194] Re-registered known tserver with Master: 5fc018b63db446139e033902fc0371ef (127.24.0.129:44999)
W20260321 08:57:45.796530 27282 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:45.798111 27282 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47330
I20260321 08:57:45.800935 27524 heartbeater.cc:499] Master 127.24.0.190:35585 was elected leader, sending a full tablet report...
W20260321 08:57:45.805728 27282 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:45.819541 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:45.820387 24578 tablet_replica.cc:333] stopping tablet replica
I20260321 08:57:45.821018 24578 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 2 LEADER]: Raft consensus shutting down.
I20260321 08:57:45.821534 24578 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 5fc018b63db446139e033902fc0371ef [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:45.841719 24578 tablet_server.cc:196] TabletServer@127.24.0.129:44999 shutdown complete.
I20260321 08:57:45.854331 24578 master.cc:562] Master@127.24.0.190:35585 shutting down...
I20260321 08:57:45.870309 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:45.870922 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:45.871291 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 54af501bc7ed4f3aa0760f627eb93acf: stopping tablet replica
I20260321 08:57:45.890182 24578 master.cc:584] Master@127.24.0.190:35585 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (724 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20260321 08:57:45.920013 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:42003
I20260321 08:57:45.921330 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:45.926790 27530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:45.927757 27531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:45.930490 27533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:45.930558 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:45.931735 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:45.931991 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:45.932188 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083465932171 us; error 0 us; skew 500 ppm
I20260321 08:57:45.932888 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:45.935415 24578 webserver.cc:492] Webserver started at http://127.24.0.190:38567/ using document root <none> and password file <none>
I20260321 08:57:45.936036 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:45.936256 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:45.936558 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:45.937911 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "5b5c7fe4df6d476a97118f6b2c7b2108"
format_stamp: "Formatted at 2026-03-21 08:57:45 on dist-test-slave-p54g"
I20260321 08:57:45.943322 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260321 08:57:45.947252 27538 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:45.948068 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:57:45.948374 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "5b5c7fe4df6d476a97118f6b2c7b2108"
format_stamp: "Formatted at 2026-03-21 08:57:45 on dist-test-slave-p54g"
I20260321 08:57:45.948719 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:45.962033 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:45.963218 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:46.005617 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:42003
I20260321 08:57:46.005677 27589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:42003 every 8 connection(s)
I20260321 08:57:46.010311 27590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:46.019760 27590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108: Bootstrap starting.
I20260321 08:57:46.023711 27590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:46.027374 27590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108: No bootstrap required, opened a new log
I20260321 08:57:46.029209 27590 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5c7fe4df6d476a97118f6b2c7b2108" member_type: VOTER }
I20260321 08:57:46.029592 27590 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:46.029794 27590 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5c7fe4df6d476a97118f6b2c7b2108, State: Initialized, Role: FOLLOWER
I20260321 08:57:46.030280 27590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5c7fe4df6d476a97118f6b2c7b2108" member_type: VOTER }
I20260321 08:57:46.030697 27590 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:46.030901 27590 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:46.031138 27590 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:46.035147 27590 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5c7fe4df6d476a97118f6b2c7b2108" member_type: VOTER }
I20260321 08:57:46.035605 27590 leader_election.cc:304] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b5c7fe4df6d476a97118f6b2c7b2108; no voters:
I20260321 08:57:46.036804 27590 leader_election.cc:290] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:46.037094 27593 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:46.038690 27593 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 1 LEADER]: Becoming Leader. State: Replica: 5b5c7fe4df6d476a97118f6b2c7b2108, State: Running, Role: LEADER
I20260321 08:57:46.039391 27593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5c7fe4df6d476a97118f6b2c7b2108" member_type: VOTER }
I20260321 08:57:46.040144 27590 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:46.042239 27594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b5c7fe4df6d476a97118f6b2c7b2108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5c7fe4df6d476a97118f6b2c7b2108" member_type: VOTER } }
I20260321 08:57:46.042339 27595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b5c7fe4df6d476a97118f6b2c7b2108. Latest consensus state: current_term: 1 leader_uuid: "5b5c7fe4df6d476a97118f6b2c7b2108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b5c7fe4df6d476a97118f6b2c7b2108" member_type: VOTER } }
I20260321 08:57:46.042799 27594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:46.042909 27595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:46.052058 27598 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:46.058749 27598 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:46.060812 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:57:46.068392 27598 catalog_manager.cc:1357] Generated new cluster ID: 5a3fade9f21e49e7bce1e7c84c924e33
I20260321 08:57:46.068679 27598 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:46.083608 27598 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:57:46.085197 27598 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:46.102804 27598 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108: Generated new TSK 0
I20260321 08:57:46.103434 27598 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:46.127097 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:46.133020 27611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:46.133932 27612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:46.138607 27614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:46.138662 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:46.139845 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:46.140136 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:46.140336 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083466140319 us; error 0 us; skew 500 ppm
I20260321 08:57:46.141021 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:46.143719 24578 webserver.cc:492] Webserver started at http://127.24.0.129:36629/ using document root <none> and password file <none>
I20260321 08:57:46.144312 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:46.144491 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:46.144758 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:46.145803 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878"
format_stamp: "Formatted at 2026-03-21 08:57:46 on dist-test-slave-p54g"
I20260321 08:57:46.150398 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260321 08:57:46.153762 27619 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:46.154531 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260321 08:57:46.154793 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878"
format_stamp: "Formatted at 2026-03-21 08:57:46 on dist-test-slave-p54g"
I20260321 08:57:46.155037 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:46.187215 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:46.188230 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:46.221769 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:46.222060 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:46.222330 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:46.222515 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:46.299607 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:38591
I20260321 08:57:46.299738 27689 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:38591 every 8 connection(s)
I20260321 08:57:46.305365 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260321 08:57:46.305676 27690 heartbeater.cc:344] Connected to a master server at 127.24.0.190:42003
I20260321 08:57:46.306171 27690 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:46.307298 27690 heartbeater.cc:507] Master 127.24.0.190:42003 requested a full tablet report, sending...
I20260321 08:57:46.309587 27555 ts_manager.cc:194] Registered new tserver with Master: 2b9b5fe6f4ff45cda1e36adbe3e5b878 (127.24.0.129:38591)
I20260321 08:57:46.311843 27555 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33578
W20260321 08:57:46.316489 27692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:46.320674 27693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:46.321930 27695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:46.322872 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:46.324069 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:46.324342 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:46.324540 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083466324523 us; error 0 us; skew 500 ppm
I20260321 08:57:46.325232 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:46.328128 24578 webserver.cc:492] Webserver started at http://127.24.0.130:42267/ using document root <none> and password file <none>
I20260321 08:57:46.328744 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:46.328984 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:46.329309 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:46.330715 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-1-root/instance:
uuid: "6b22d084982541748515af60f41b0598"
format_stamp: "Formatted at 2026-03-21 08:57:46 on dist-test-slave-p54g"
I20260321 08:57:46.336505 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260321 08:57:46.340787 27700 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:46.344576 24578 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260321 08:57:46.344918 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-1-root
uuid: "6b22d084982541748515af60f41b0598"
format_stamp: "Formatted at 2026-03-21 08:57:46 on dist-test-slave-p54g"
I20260321 08:57:46.345278 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:46.363345 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:46.364708 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:46.370486 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:46.370779 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:46.371105 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:46.371369 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:46.437520 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.130:45423
I20260321 08:57:46.437656 27769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.130:45423 every 8 connection(s)
I20260321 08:57:46.442340 27770 heartbeater.cc:344] Connected to a master server at 127.24.0.190:42003
I20260321 08:57:46.442775 27770 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:46.443681 27770 heartbeater.cc:507] Master 127.24.0.190:42003 requested a full tablet report, sending...
I20260321 08:57:46.445822 27555 ts_manager.cc:194] Registered new tserver with Master: 6b22d084982541748515af60f41b0598 (127.24.0.130:45423)
I20260321 08:57:46.446693 24578 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.006099982s
I20260321 08:57:46.447690 27555 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33584
I20260321 08:57:46.479033 27655 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260321 08:57:46.480170 27655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:46.490779 27780 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Bootstrap starting.
I20260321 08:57:46.494980 27780 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:46.498844 27780 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: No bootstrap required, opened a new log
I20260321 08:57:46.499198 27780 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260321 08:57:46.501055 27780 raft_consensus.cc:359] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } }
I20260321 08:57:46.501468 27780 raft_consensus.cc:385] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:46.501679 27780 raft_consensus.cc:740] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9b5fe6f4ff45cda1e36adbe3e5b878, State: Initialized, Role: FOLLOWER
I20260321 08:57:46.502197 27780 consensus_queue.cc:260] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } }
I20260321 08:57:46.502651 27780 raft_consensus.cc:399] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:46.502864 27780 raft_consensus.cc:493] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:46.503089 27780 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:46.507053 27780 raft_consensus.cc:515] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } }
I20260321 08:57:46.507537 27780 leader_election.cc:304] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b9b5fe6f4ff45cda1e36adbe3e5b878; no voters:
I20260321 08:57:46.508821 27780 leader_election.cc:290] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:46.509140 27782 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:46.510632 27690 heartbeater.cc:499] Master 127.24.0.190:42003 was elected leader, sending a full tablet report...
I20260321 08:57:46.511292 27780 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260321 08:57:46.511840 27782 raft_consensus.cc:697] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 1 LEADER]: Becoming Leader. State: Replica: 2b9b5fe6f4ff45cda1e36adbe3e5b878, State: Running, Role: LEADER
I20260321 08:57:46.512482 27782 consensus_queue.cc:237] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } }
I20260321 08:57:46.515669 27784 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: TxnStatusTablet state changed. Reason: New leader 2b9b5fe6f4ff45cda1e36adbe3e5b878. Latest consensus state: current_term: 1 leader_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" committed_config { opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } } }
I20260321 08:57:46.516194 27784 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: This TxnStatusTablet replica's current role is: LEADER
I20260321 08:57:46.516328 27783 tablet_replica.cc:442] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" committed_config { opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } } }
I20260321 08:57:46.516894 27783 tablet_replica.cc:445] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: This TxnStatusTablet replica's current role is: LEADER
W20260321 08:57:46.519034 27555 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:46.522889 27786 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260321 08:57:46.523299 27786 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260321 08:57:46.525523 27555 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:46.575965 27791 ts_tablet_manager.cc:933] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: Initiating tablet copy from peer 2b9b5fe6f4ff45cda1e36adbe3e5b878 (127.24.0.129:38591)
I20260321 08:57:46.577510 27791 tablet_copy_client.cc:323] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: tablet copy: Beginning tablet copy session from remote peer at address 127.24.0.129:38591
I20260321 08:57:46.585347 27675 tablet_copy_service.cc:140] P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 6b22d084982541748515af60f41b0598 ({username='slave'} at 127.0.0.1:57140)
I20260321 08:57:46.585779 27675 tablet_copy_service.cc:161] P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 6b22d084982541748515af60f41b0598 at {username='slave'} at 127.0.0.1:57140: session id = 6b22d084982541748515af60f41b0598-11111111111111111111111111111111
I20260321 08:57:46.590446 27675 tablet_copy_source_session.cc:215] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 08:57:46.593005 27791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:46.602640 27791 tablet_copy_client.cc:806] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: tablet copy: Starting download of 0 data blocks...
I20260321 08:57:46.603085 27791 tablet_copy_client.cc:670] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: tablet copy: Starting download of 1 WAL segments...
I20260321 08:57:46.606333 27791 tablet_copy_client.cc:538] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 08:57:46.611768 27791 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: Bootstrap starting.
I20260321 08:57:46.631635 27791 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 08:57:46.632272 27791 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: Bootstrap complete.
I20260321 08:57:46.632779 27791 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.004s
I20260321 08:57:46.634550 27791 raft_consensus.cc:359] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } }
I20260321 08:57:46.634930 27791 raft_consensus.cc:740] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 6b22d084982541748515af60f41b0598, State: Initialized, Role: NON_PARTICIPANT
I20260321 08:57:46.635411 27791 consensus_queue.cc:260] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } }
I20260321 08:57:46.637212 27770 heartbeater.cc:499] Master 127.24.0.190:42003 was elected leader, sending a full tablet report...
I20260321 08:57:46.637064 27794 tablet_replica.cc:442] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } } }
I20260321 08:57:46.637776 27794 tablet_replica.cc:445] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20260321 08:57:46.637871 27791 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260321 08:57:46.639460 27675 tablet_copy_service.cc:342] P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Request end of tablet copy session 6b22d084982541748515af60f41b0598-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:57140
I20260321 08:57:46.639842 27675 tablet_copy_service.cc:434] P 2b9b5fe6f4ff45cda1e36adbe3e5b878: ending tablet copy session 6b22d084982541748515af60f41b0598-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 6b22d084982541748515af60f41b0598
W20260321 08:57:46.640060 27555 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260321 08:57:46.643492 27555 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:46.667629 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:57:46.684592 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:46.685400 24578 tablet_replica.cc:333] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: stopping tablet replica
I20260321 08:57:46.686033 24578 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:46.686589 24578 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:46.707989 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:57:46.718657 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:46.725529 27795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:46.726042 27796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:46.727298 27798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:46.729518 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:46.730794 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:46.731060 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:46.731251 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083466731235 us; error 0 us; skew 500 ppm
I20260321 08:57:46.731951 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:46.734608 24578 webserver.cc:492] Webserver started at http://127.24.0.129:36629/ using document root <none> and password file <none>
I20260321 08:57:46.735190 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:46.735399 24578 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260321 08:57:46.739744 24578 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260321 08:57:46.743037 27803 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:46.743929 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:57:46.744370 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878"
format_stamp: "Formatted at 2026-03-21 08:57:46 on dist-test-slave-p54g"
I20260321 08:57:46.744844 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:46.758882 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:46.760286 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:46.767128 27812 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20260321 08:57:46.774070 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260321 08:57:46.774562 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20260321 08:57:46.774901 24578 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260321 08:57:46.780857 24578 ts_tablet_manager.cc:616] Registered 1 tablets
I20260321 08:57:46.781096 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s user 0.006s sys 0.000s
I20260321 08:57:46.781157 27812 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Bootstrap starting.
I20260321 08:57:46.814009 27812 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 08:57:46.814996 27812 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Bootstrap complete.
I20260321 08:57:46.815625 27812 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.012s
I20260321 08:57:46.818223 27812 raft_consensus.cc:359] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } }
I20260321 08:57:46.818763 27812 raft_consensus.cc:740] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b9b5fe6f4ff45cda1e36adbe3e5b878, State: Initialized, Role: FOLLOWER
I20260321 08:57:46.819497 27812 consensus_queue.cc:260] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } }
I20260321 08:57:46.820156 27812 raft_consensus.cc:399] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:46.820443 27812 raft_consensus.cc:493] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:46.820760 27812 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:46.827178 27812 raft_consensus.cc:515] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } }
I20260321 08:57:46.827833 27812 leader_election.cc:304] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b9b5fe6f4ff45cda1e36adbe3e5b878; no voters:
I20260321 08:57:46.830437 27812 leader_election.cc:290] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260321 08:57:46.830808 27823 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 08:57:46.839229 27812 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878: Time spent starting tablet: real 0.023s user 0.017s sys 0.004s
I20260321 08:57:46.840404 27823 raft_consensus.cc:697] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 2 LEADER]: Becoming Leader. State: Replica: 2b9b5fe6f4ff45cda1e36adbe3e5b878, State: Running, Role: LEADER
I20260321 08:57:46.841424 27823 consensus_queue.cc:237] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } }
I20260321 08:57:46.844805 27825 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" committed_config { opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } } }
I20260321 08:57:46.845293 27825 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260321 08:57:46.857903 27826 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 2b9b5fe6f4ff45cda1e36adbe3e5b878. Latest consensus state: current_term: 2 leader_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" committed_config { opid_index: -1 peers { permanent_uuid: "2b9b5fe6f4ff45cda1e36adbe3e5b878" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 38591 } } }
I20260321 08:57:46.858448 27826 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260321 08:57:46.858878 27832 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260321 08:57:46.859292 27832 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260321 08:57:46.893802 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:38591
I20260321 08:57:46.893949 27880 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:38591 every 8 connection(s)
I20260321 08:57:46.899365 27881 heartbeater.cc:344] Connected to a master server at 127.24.0.190:42003
I20260321 08:57:46.900008 24578 tablet_server.cc:179] TabletServer@127.24.0.129:38591 shutting down...
I20260321 08:57:46.900139 27881 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:46.902876 27881 heartbeater.cc:507] Master 127.24.0.190:42003 requested a full tablet report, sending...
I20260321 08:57:46.906054 27555 ts_manager.cc:194] Re-registered known tserver with Master: 2b9b5fe6f4ff45cda1e36adbe3e5b878 (127.24.0.129:38591)
W20260321 08:57:46.906558 27555 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:46.908077 27555 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33632
I20260321 08:57:46.912796 27881 heartbeater.cc:499] Master 127.24.0.190:42003 was elected leader, sending a full tablet report...
W20260321 08:57:46.917878 27555 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:46.922708 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:46.923377 24578 tablet_replica.cc:333] stopping tablet replica
I20260321 08:57:46.923866 24578 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 2 LEADER]: Raft consensus shutting down.
I20260321 08:57:46.924275 24578 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 2b9b5fe6f4ff45cda1e36adbe3e5b878 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:46.934458 24578 tablet_server.cc:196] TabletServer@127.24.0.129:38591 shutdown complete.
I20260321 08:57:46.944393 24578 tablet_server.cc:179] TabletServer@127.24.0.130:0 shutting down...
I20260321 08:57:46.964897 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:46.965603 24578 tablet_replica.cc:333] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598: stopping tablet replica
I20260321 08:57:46.966156 24578 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260321 08:57:46.966547 24578 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 6b22d084982541748515af60f41b0598 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260321 08:57:46.985385 24578 tablet_server.cc:196] TabletServer@127.24.0.130:0 shutdown complete.
I20260321 08:57:46.995550 24578 master.cc:562] Master@127.24.0.190:42003 shutting down...
I20260321 08:57:47.011802 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:47.012965 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:47.013506 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 5b5c7fe4df6d476a97118f6b2c7b2108: stopping tablet replica
I20260321 08:57:47.032423 24578 master.cc:584] Master@127.24.0.190:42003 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1137 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20260321 08:57:47.056697 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:33635
I20260321 08:57:47.057724 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:47.063359 27887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:47.064446 27888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:47.065940 27890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:47.068225 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:47.068934 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:47.069104 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:47.069245 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083467069231 us; error 0 us; skew 500 ppm
I20260321 08:57:47.069751 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:47.072335 24578 webserver.cc:492] Webserver started at http://127.24.0.190:35611/ using document root <none> and password file <none>
I20260321 08:57:47.072795 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:47.072954 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:47.073190 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:47.074201 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "426fed133e704c66a5ee4a4ffa9ac320"
format_stamp: "Formatted at 2026-03-21 08:57:47 on dist-test-slave-p54g"
I20260321 08:57:47.078447 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260321 08:57:47.081655 27895 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:47.082464 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260321 08:57:47.082803 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "426fed133e704c66a5ee4a4ffa9ac320"
format_stamp: "Formatted at 2026-03-21 08:57:47 on dist-test-slave-p54g"
I20260321 08:57:47.083150 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:47.114661 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:47.116003 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:47.159060 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:33635
I20260321 08:57:47.159119 27946 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:33635 every 8 connection(s)
I20260321 08:57:47.163762 27947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:47.173185 27947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320: Bootstrap starting.
I20260321 08:57:47.177150 27947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:47.180835 27947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320: No bootstrap required, opened a new log
I20260321 08:57:47.182601 27947 raft_consensus.cc:359] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426fed133e704c66a5ee4a4ffa9ac320" member_type: VOTER }
I20260321 08:57:47.182958 27947 raft_consensus.cc:385] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:47.183161 27947 raft_consensus.cc:740] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426fed133e704c66a5ee4a4ffa9ac320, State: Initialized, Role: FOLLOWER
I20260321 08:57:47.183676 27947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426fed133e704c66a5ee4a4ffa9ac320" member_type: VOTER }
I20260321 08:57:47.184132 27947 raft_consensus.cc:399] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:47.184340 27947 raft_consensus.cc:493] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:47.184572 27947 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:47.188591 27947 raft_consensus.cc:515] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426fed133e704c66a5ee4a4ffa9ac320" member_type: VOTER }
I20260321 08:57:47.189035 27947 leader_election.cc:304] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 426fed133e704c66a5ee4a4ffa9ac320; no voters:
I20260321 08:57:47.190171 27947 leader_election.cc:290] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:47.190469 27950 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:47.192056 27950 raft_consensus.cc:697] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 1 LEADER]: Becoming Leader. State: Replica: 426fed133e704c66a5ee4a4ffa9ac320, State: Running, Role: LEADER
I20260321 08:57:47.192750 27950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426fed133e704c66a5ee4a4ffa9ac320" member_type: VOTER }
I20260321 08:57:47.193504 27947 sys_catalog.cc:565] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:47.195484 27951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "426fed133e704c66a5ee4a4ffa9ac320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426fed133e704c66a5ee4a4ffa9ac320" member_type: VOTER } }
I20260321 08:57:47.195562 27952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 426fed133e704c66a5ee4a4ffa9ac320. Latest consensus state: current_term: 1 leader_uuid: "426fed133e704c66a5ee4a4ffa9ac320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "426fed133e704c66a5ee4a4ffa9ac320" member_type: VOTER } }
I20260321 08:57:47.196081 27951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:47.196152 27952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:47.212819 27955 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:47.219197 27955 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:47.221473 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:57:47.228968 27955 catalog_manager.cc:1357] Generated new cluster ID: eb0dd047fa074864a58f8ece9e4a5aad
I20260321 08:57:47.229249 27955 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:47.257359 27955 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:57:47.258976 27955 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:47.276831 27955 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320: Generated new TSK 0
I20260321 08:57:47.277421 27955 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:47.287323 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:47.292879 27968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:47.293819 27969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:47.296712 27971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:47.297269 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:47.297952 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:47.298131 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:47.298280 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083467298263 us; error 0 us; skew 500 ppm
I20260321 08:57:47.298734 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:47.301168 24578 webserver.cc:492] Webserver started at http://127.24.0.129:36677/ using document root <none> and password file <none>
I20260321 08:57:47.301605 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:47.301774 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:47.302011 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:47.302958 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "6364e27be8c74c779c219a51105c6400"
format_stamp: "Formatted at 2026-03-21 08:57:47 on dist-test-slave-p54g"
I20260321 08:57:47.307096 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260321 08:57:47.310189 27976 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:47.310940 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260321 08:57:47.311285 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "6364e27be8c74c779c219a51105c6400"
format_stamp: "Formatted at 2026-03-21 08:57:47 on dist-test-slave-p54g"
I20260321 08:57:47.311628 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:47.327302 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:47.328573 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:47.340312 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:47.340590 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:47.340869 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:47.341074 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:47.411000 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:33103
I20260321 08:57:47.411069 28046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:33103 every 8 connection(s)
I20260321 08:57:47.416746 28047 heartbeater.cc:344] Connected to a master server at 127.24.0.190:33635
I20260321 08:57:47.417089 28047 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:47.417783 28047 heartbeater.cc:507] Master 127.24.0.190:33635 requested a full tablet report, sending...
I20260321 08:57:47.419759 27912 ts_manager.cc:194] Registered new tserver with Master: 6364e27be8c74c779c219a51105c6400 (127.24.0.129:33103)
I20260321 08:57:47.420199 24578 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004988741s
I20260321 08:57:47.422040 27912 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38970
I20260321 08:57:47.457875 28012 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260321 08:57:47.458957 28012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:47.469440 28057 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400: Bootstrap starting.
I20260321 08:57:47.473546 28057 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:47.477363 28057 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400: No bootstrap required, opened a new log
I20260321 08:57:47.477703 28057 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260321 08:57:47.479537 28057 raft_consensus.cc:359] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "6364e27be8c74c779c219a51105c6400" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33103 } }
I20260321 08:57:47.479944 28057 raft_consensus.cc:385] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:47.480163 28057 raft_consensus.cc:740] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6364e27be8c74c779c219a51105c6400, State: Initialized, Role: FOLLOWER
I20260321 08:57:47.480654 28057 consensus_queue.cc:260] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "6364e27be8c74c779c219a51105c6400" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33103 } }
I20260321 08:57:47.481154 28057 raft_consensus.cc:399] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:47.481384 28057 raft_consensus.cc:493] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:47.481619 28057 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:47.485893 28057 raft_consensus.cc:515] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "6364e27be8c74c779c219a51105c6400" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33103 } }
I20260321 08:57:47.486358 28057 leader_election.cc:304] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6364e27be8c74c779c219a51105c6400; no voters:
I20260321 08:57:47.487546 28057 leader_election.cc:290] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:47.487928 28059 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:47.489296 28047 heartbeater.cc:499] Master 127.24.0.190:33635 was elected leader, sending a full tablet report...
I20260321 08:57:47.489910 28057 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260321 08:57:47.490471 28059 raft_consensus.cc:697] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 1 LEADER]: Becoming Leader. State: Replica: 6364e27be8c74c779c219a51105c6400, State: Running, Role: LEADER
I20260321 08:57:47.491174 28059 consensus_queue.cc:237] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "6364e27be8c74c779c219a51105c6400" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33103 } }
I20260321 08:57:47.494459 28061 tablet_replica.cc:442] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400: TxnStatusTablet state changed. Reason: New leader 6364e27be8c74c779c219a51105c6400. Latest consensus state: current_term: 1 leader_uuid: "6364e27be8c74c779c219a51105c6400" committed_config { opid_index: -1 peers { permanent_uuid: "6364e27be8c74c779c219a51105c6400" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33103 } } }
I20260321 08:57:47.494951 28061 tablet_replica.cc:445] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400: This TxnStatusTablet replica's current role is: LEADER
I20260321 08:57:47.495806 28060 tablet_replica.cc:442] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6364e27be8c74c779c219a51105c6400" committed_config { opid_index: -1 peers { permanent_uuid: "6364e27be8c74c779c219a51105c6400" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33103 } } }
I20260321 08:57:47.496331 28060 tablet_replica.cc:445] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400: This TxnStatusTablet replica's current role is: LEADER
W20260321 08:57:47.496780 27912 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:47.500773 28063 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260321 08:57:47.501158 28063 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260321 08:57:47.502892 27912 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260321 08:57:47.550967 27992 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260321 08:57:47.551322 27992 txn_status_manager.cc:244] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260321 08:57:47.551684 27992 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20260321 08:57:47.559561 27992 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260321 08:57:47.598290 27992 txn_status_manager.cc:244] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260321 08:57:47.598742 27992 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20260321 08:57:47.606052 27992 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20260321 08:57:47.618204 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39030
W20260321 08:57:47.620322 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39030
W20260321 08:57:47.622182 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39030
W20260321 08:57:47.624061 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39030
W20260321 08:57:47.625988 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39030
W20260321 08:57:47.627821 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39030
W20260321 08:57:47.637684 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39034
W20260321 08:57:47.639739 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39034
W20260321 08:57:47.641721 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39034
W20260321 08:57:47.643497 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39034
W20260321 08:57:47.645263 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39034
W20260321 08:57:47.647120 28011 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39034
I20260321 08:57:47.653378 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:57:47.671094 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:47.671929 24578 tablet_replica.cc:333] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400: stopping tablet replica
I20260321 08:57:47.672549 24578 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:47.673097 24578 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 6364e27be8c74c779c219a51105c6400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:47.695498 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:57:47.709197 24578 master.cc:562] Master@127.24.0.190:33635 shutting down...
I20260321 08:57:47.727257 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:47.727756 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:47.728222 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 426fed133e704c66a5ee4a4ffa9ac320: stopping tablet replica
I20260321 08:57:47.735994 24578 master.cc:584] Master@127.24.0.190:33635 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (703 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20260321 08:57:47.759951 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:46537
I20260321 08:57:47.760978 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:47.765813 28072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:47.767825 28073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:47.768718 28075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:47.768882 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:47.769732 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:47.769915 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:47.770056 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083467770039 us; error 0 us; skew 500 ppm
I20260321 08:57:47.770596 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:47.772913 24578 webserver.cc:492] Webserver started at http://127.24.0.190:37049/ using document root <none> and password file <none>
I20260321 08:57:47.773362 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:47.773542 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:47.773782 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:47.774827 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "7bbd7b5ba1b54e8d87a8ba0d6cd2646f"
format_stamp: "Formatted at 2026-03-21 08:57:47 on dist-test-slave-p54g"
I20260321 08:57:47.779306 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260321 08:57:47.782816 28080 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:47.783726 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260321 08:57:47.784026 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "7bbd7b5ba1b54e8d87a8ba0d6cd2646f"
format_stamp: "Formatted at 2026-03-21 08:57:47 on dist-test-slave-p54g"
I20260321 08:57:47.784298 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:47.807173 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:47.808197 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:47.854911 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:46537
I20260321 08:57:47.855047 28131 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:46537 every 8 connection(s)
I20260321 08:57:47.858839 28132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:47.872577 28132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f: Bootstrap starting.
I20260321 08:57:47.876933 28132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:47.880548 28132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f: No bootstrap required, opened a new log
I20260321 08:57:47.882344 28132 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbd7b5ba1b54e8d87a8ba0d6cd2646f" member_type: VOTER }
I20260321 08:57:47.882717 28132 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:47.882891 28132 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bbd7b5ba1b54e8d87a8ba0d6cd2646f, State: Initialized, Role: FOLLOWER
I20260321 08:57:47.883323 28132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbd7b5ba1b54e8d87a8ba0d6cd2646f" member_type: VOTER }
I20260321 08:57:47.883718 28132 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:47.883926 28132 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:47.884145 28132 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:47.888355 28132 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbd7b5ba1b54e8d87a8ba0d6cd2646f" member_type: VOTER }
I20260321 08:57:47.888797 28132 leader_election.cc:304] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bbd7b5ba1b54e8d87a8ba0d6cd2646f; no voters:
I20260321 08:57:47.889843 28132 leader_election.cc:290] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:47.890128 28135 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:47.891330 28135 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 1 LEADER]: Becoming Leader. State: Replica: 7bbd7b5ba1b54e8d87a8ba0d6cd2646f, State: Running, Role: LEADER
I20260321 08:57:47.891970 28135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbd7b5ba1b54e8d87a8ba0d6cd2646f" member_type: VOTER }
I20260321 08:57:47.892540 28132 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:47.894490 28137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bbd7b5ba1b54e8d87a8ba0d6cd2646f. Latest consensus state: current_term: 1 leader_uuid: "7bbd7b5ba1b54e8d87a8ba0d6cd2646f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbd7b5ba1b54e8d87a8ba0d6cd2646f" member_type: VOTER } }
I20260321 08:57:47.895128 28137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:47.894471 28136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bbd7b5ba1b54e8d87a8ba0d6cd2646f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbd7b5ba1b54e8d87a8ba0d6cd2646f" member_type: VOTER } }
I20260321 08:57:47.895959 28136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:47.898578 28140 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:47.902627 28140 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:47.911156 28140 catalog_manager.cc:1357] Generated new cluster ID: 272633f20ce84e439ea8ed11733e989d
I20260321 08:57:47.911466 28140 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:47.911721 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:57:47.951486 28140 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:57:47.952695 28140 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:47.968298 28140 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f: Generated new TSK 0
I20260321 08:57:47.968828 28140 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:47.978225 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:47.983204 28153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:47.984241 28154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:47.986339 28156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:47.986469 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:47.987313 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:47.987491 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:47.987634 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083467987619 us; error 0 us; skew 500 ppm
I20260321 08:57:47.988158 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:47.990223 24578 webserver.cc:492] Webserver started at http://127.24.0.129:44287/ using document root <none> and password file <none>
I20260321 08:57:47.990658 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:47.990813 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:47.991048 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:47.992081 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "ca069028edbc4541aabb73974ab7322f"
format_stamp: "Formatted at 2026-03-21 08:57:47 on dist-test-slave-p54g"
I20260321 08:57:47.996109 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260321 08:57:47.999096 28161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:47.999733 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260321 08:57:48.000005 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "ca069028edbc4541aabb73974ab7322f"
format_stamp: "Formatted at 2026-03-21 08:57:47 on dist-test-slave-p54g"
I20260321 08:57:48.000278 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:48.014880 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:48.015786 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:48.020146 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:48.020447 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.020731 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:48.020942 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.065824 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:33145
I20260321 08:57:48.065941 28231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:33145 every 8 connection(s)
I20260321 08:57:48.070154 28232 heartbeater.cc:344] Connected to a master server at 127.24.0.190:46537
I20260321 08:57:48.070523 28232 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:48.071147 28232 heartbeater.cc:507] Master 127.24.0.190:46537 requested a full tablet report, sending...
I20260321 08:57:48.072986 28097 ts_manager.cc:194] Registered new tserver with Master: ca069028edbc4541aabb73974ab7322f (127.24.0.129:33145)
I20260321 08:57:48.073602 24578 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004837825s
I20260321 08:57:48.074572 28097 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54790
I20260321 08:57:48.102566 28197 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260321 08:57:48.103623 28197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:48.114203 28242 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f: Bootstrap starting.
I20260321 08:57:48.118098 28242 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:48.121583 28242 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f: No bootstrap required, opened a new log
I20260321 08:57:48.121929 28242 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260321 08:57:48.123637 28242 raft_consensus.cc:359] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "ca069028edbc4541aabb73974ab7322f" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33145 } }
I20260321 08:57:48.124034 28242 raft_consensus.cc:385] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:48.124234 28242 raft_consensus.cc:740] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca069028edbc4541aabb73974ab7322f, State: Initialized, Role: FOLLOWER
I20260321 08:57:48.124727 28242 consensus_queue.cc:260] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ca069028edbc4541aabb73974ab7322f" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33145 } }
I20260321 08:57:48.125156 28242 raft_consensus.cc:399] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:48.125362 28242 raft_consensus.cc:493] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:48.125607 28242 raft_consensus.cc:3060] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:48.129580 28242 raft_consensus.cc:515] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "ca069028edbc4541aabb73974ab7322f" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33145 } }
I20260321 08:57:48.130040 28242 leader_election.cc:304] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca069028edbc4541aabb73974ab7322f; no voters:
I20260321 08:57:48.131031 28242 leader_election.cc:290] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:48.131328 28244 raft_consensus.cc:2804] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:48.132665 28244 raft_consensus.cc:697] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 1 LEADER]: Becoming Leader. State: Replica: ca069028edbc4541aabb73974ab7322f, State: Running, Role: LEADER
I20260321 08:57:48.132718 28232 heartbeater.cc:499] Master 127.24.0.190:46537 was elected leader, sending a full tablet report...
I20260321 08:57:48.133381 28242 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260321 08:57:48.133401 28244 consensus_queue.cc:237] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ca069028edbc4541aabb73974ab7322f" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33145 } }
I20260321 08:57:48.136996 28246 tablet_replica.cc:442] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f: TxnStatusTablet state changed. Reason: New leader ca069028edbc4541aabb73974ab7322f. Latest consensus state: current_term: 1 leader_uuid: "ca069028edbc4541aabb73974ab7322f" committed_config { opid_index: -1 peers { permanent_uuid: "ca069028edbc4541aabb73974ab7322f" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33145 } } }
I20260321 08:57:48.137528 28246 tablet_replica.cc:445] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f: This TxnStatusTablet replica's current role is: LEADER
I20260321 08:57:48.137784 28245 tablet_replica.cc:442] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca069028edbc4541aabb73974ab7322f" committed_config { opid_index: -1 peers { permanent_uuid: "ca069028edbc4541aabb73974ab7322f" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33145 } } }
I20260321 08:57:48.139767 28245 tablet_replica.cc:445] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f: This TxnStatusTablet replica's current role is: LEADER
W20260321 08:57:48.139909 28097 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:48.140468 28249 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260321 08:57:48.140810 28249 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260321 08:57:48.143360 28097 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260321 08:57:48.160383 28197 tablet_service.cc:1511] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260321 08:57:48.161428 28197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:48.170899 28242 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f: Bootstrap starting.
I20260321 08:57:48.174822 28242 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:48.178427 28242 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f: No bootstrap required, opened a new log
I20260321 08:57:48.178787 28242 ts_tablet_manager.cc:1403] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260321 08:57:48.180527 28242 raft_consensus.cc:359] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "ca069028edbc4541aabb73974ab7322f" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33145 } }
I20260321 08:57:48.180908 28242 raft_consensus.cc:385] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:48.181106 28242 raft_consensus.cc:740] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca069028edbc4541aabb73974ab7322f, State: Initialized, Role: FOLLOWER
I20260321 08:57:48.181555 28242 consensus_queue.cc:260] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ca069028edbc4541aabb73974ab7322f" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33145 } }
I20260321 08:57:48.181967 28242 raft_consensus.cc:399] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:48.182178 28242 raft_consensus.cc:493] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:48.182410 28242 raft_consensus.cc:3060] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:48.186668 28242 raft_consensus.cc:515] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "ca069028edbc4541aabb73974ab7322f" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33145 } }
I20260321 08:57:48.187131 28242 leader_election.cc:304] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca069028edbc4541aabb73974ab7322f; no voters:
I20260321 08:57:48.187574 28242 leader_election.cc:290] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:48.187716 28245 raft_consensus.cc:2804] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:48.188232 28245 raft_consensus.cc:697] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 1 LEADER]: Becoming Leader. State: Replica: ca069028edbc4541aabb73974ab7322f, State: Running, Role: LEADER
I20260321 08:57:48.188755 28245 consensus_queue.cc:237] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ca069028edbc4541aabb73974ab7322f" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 33145 } }
I20260321 08:57:48.189376 28242 ts_tablet_manager.cc:1434] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
W20260321 08:57:48.193750 28097 catalog_manager.cc:5504] Ignoring report from unknown tablet 22222222222222222222222222222222
W20260321 08:57:48.197366 28097 catalog_manager.cc:5504] Ignoring report from unknown tablet 22222222222222222222222222222222
I20260321 08:57:48.218546 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:57:48.235774 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:48.236563 24578 tablet_replica.cc:333] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f: stopping tablet replica
I20260321 08:57:48.237039 24578 raft_consensus.cc:2243] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:48.237460 24578 raft_consensus.cc:2272] T 22222222222222222222222222222222 P ca069028edbc4541aabb73974ab7322f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:48.239763 24578 tablet_replica.cc:333] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f: stopping tablet replica
I20260321 08:57:48.240237 24578 raft_consensus.cc:2243] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:48.240626 24578 raft_consensus.cc:2272] T 11111111111111111111111111111111 P ca069028edbc4541aabb73974ab7322f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:48.251709 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:57:48.261794 24578 master.cc:562] Master@127.24.0.190:46537 shutting down...
I20260321 08:57:48.274551 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:48.275023 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:48.275432 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7bbd7b5ba1b54e8d87a8ba0d6cd2646f: stopping tablet replica
I20260321 08:57:48.292973 24578 master.cc:584] Master@127.24.0.190:46537 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (555 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3121 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20260321 08:57:48.315223 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:35185
I20260321 08:57:48.316243 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:48.320587 28258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:48.321810 28259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:48.322329 28261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:48.323442 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:48.324081 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:48.324254 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:48.324369 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083468324358 us; error 0 us; skew 500 ppm
I20260321 08:57:48.324808 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:48.326803 24578 webserver.cc:492] Webserver started at http://127.24.0.190:41393/ using document root <none> and password file <none>
I20260321 08:57:48.327209 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:48.327348 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:48.327548 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:48.328593 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "68be4e29ce6b4b098455cd14a55203f5"
format_stamp: "Formatted at 2026-03-21 08:57:48 on dist-test-slave-p54g"
I20260321 08:57:48.332557 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260321 08:57:48.335585 28266 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.336272 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260321 08:57:48.336532 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "68be4e29ce6b4b098455cd14a55203f5"
format_stamp: "Formatted at 2026-03-21 08:57:48 on dist-test-slave-p54g"
I20260321 08:57:48.336764 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:48.346143 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:48.346995 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:48.379309 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:35185
I20260321 08:57:48.379420 28317 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:35185 every 8 connection(s)
I20260321 08:57:48.382740 28318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:48.391973 28318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5: Bootstrap starting.
I20260321 08:57:48.395944 28318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:48.399698 28318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5: No bootstrap required, opened a new log
I20260321 08:57:48.417791 28318 raft_consensus.cc:359] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be4e29ce6b4b098455cd14a55203f5" member_type: VOTER }
I20260321 08:57:48.418455 28318 raft_consensus.cc:385] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:48.418684 28318 raft_consensus.cc:740] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68be4e29ce6b4b098455cd14a55203f5, State: Initialized, Role: FOLLOWER
I20260321 08:57:48.419245 28318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be4e29ce6b4b098455cd14a55203f5" member_type: VOTER }
I20260321 08:57:48.419703 28318 raft_consensus.cc:399] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:48.419967 28318 raft_consensus.cc:493] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:48.420226 28318 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:48.426250 28318 raft_consensus.cc:515] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be4e29ce6b4b098455cd14a55203f5" member_type: VOTER }
I20260321 08:57:48.426724 28318 leader_election.cc:304] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68be4e29ce6b4b098455cd14a55203f5; no voters:
I20260321 08:57:48.427811 28318 leader_election.cc:290] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:48.428223 28322 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:48.429425 28322 raft_consensus.cc:697] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 1 LEADER]: Becoming Leader. State: Replica: 68be4e29ce6b4b098455cd14a55203f5, State: Running, Role: LEADER
I20260321 08:57:48.430157 28322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be4e29ce6b4b098455cd14a55203f5" member_type: VOTER }
I20260321 08:57:48.431263 28318 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:48.437095 28324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68be4e29ce6b4b098455cd14a55203f5. Latest consensus state: current_term: 1 leader_uuid: "68be4e29ce6b4b098455cd14a55203f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be4e29ce6b4b098455cd14a55203f5" member_type: VOTER } }
I20260321 08:57:48.437201 28323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68be4e29ce6b4b098455cd14a55203f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68be4e29ce6b4b098455cd14a55203f5" member_type: VOTER } }
I20260321 08:57:48.437917 28323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:48.437911 28324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:48.440546 28330 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:48.447459 28330 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:48.448745 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:57:48.456467 28330 catalog_manager.cc:1357] Generated new cluster ID: 3ad1fe67148e4582b7bfc5edecc6ee68
I20260321 08:57:48.456763 28330 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:48.480394 28330 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:57:48.481680 28330 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:48.496856 28330 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5: Generated new TSK 0
I20260321 08:57:48.497439 28330 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:48.515337 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:48.523239 28341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:48.523159 28340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:48.526461 24578 server_base.cc:1061] running on GCE node
W20260321 08:57:48.527143 28343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:48.528077 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:48.528298 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:48.528435 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083468528423 us; error 0 us; skew 500 ppm
I20260321 08:57:48.528921 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:48.531431 24578 webserver.cc:492] Webserver started at http://127.24.0.129:44299/ using document root <none> and password file <none>
I20260321 08:57:48.531909 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:48.532091 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:48.532342 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:48.533710 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "00ed22d0c0af49e4a72844c10c323d55"
format_stamp: "Formatted at 2026-03-21 08:57:48 on dist-test-slave-p54g"
I20260321 08:57:48.538187 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260321 08:57:48.541767 28348 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.542443 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260321 08:57:48.542702 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "00ed22d0c0af49e4a72844c10c323d55"
format_stamp: "Formatted at 2026-03-21 08:57:48 on dist-test-slave-p54g"
I20260321 08:57:48.542955 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:48.577275 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:48.578496 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:48.580019 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:48.582352 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:48.582553 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.582772 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:48.582927 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.621727 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:43537
I20260321 08:57:48.621832 28410 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:43537 every 8 connection(s)
I20260321 08:57:48.626217 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:48.633574 28415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:48.634548 28416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:48.636296 24578 server_base.cc:1061] running on GCE node
W20260321 08:57:48.637842 28418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:48.638504 28411 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35185
I20260321 08:57:48.638746 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260321 08:57:48.638839 28411 heartbeater.cc:461] Registering TS with master...
W20260321 08:57:48.638988 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:48.639191 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083468639174 us; error 0 us; skew 500 ppm
I20260321 08:57:48.639602 28411 heartbeater.cc:507] Master 127.24.0.190:35185 requested a full tablet report, sending...
I20260321 08:57:48.639767 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:48.641731 28283 ts_manager.cc:194] Registered new tserver with Master: 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537)
I20260321 08:57:48.642526 24578 webserver.cc:492] Webserver started at http://127.24.0.130:35763/ using document root <none> and password file <none>
I20260321 08:57:48.642998 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:48.643167 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:48.643404 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:48.643532 28283 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59098
I20260321 08:57:48.644783 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-1-root/instance:
uuid: "bd4c7686acae461fb1d92c17ccb3ce01"
format_stamp: "Formatted at 2026-03-21 08:57:48 on dist-test-slave-p54g"
I20260321 08:57:48.649230 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260321 08:57:48.652343 28423 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.653053 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260321 08:57:48.653318 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-1-root
uuid: "bd4c7686acae461fb1d92c17ccb3ce01"
format_stamp: "Formatted at 2026-03-21 08:57:48 on dist-test-slave-p54g"
I20260321 08:57:48.653596 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:48.667390 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:48.668601 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:48.669937 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:48.672086 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:48.672263 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.672442 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:48.672577 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.711191 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.130:33305
I20260321 08:57:48.711308 28485 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.130:33305 every 8 connection(s)
I20260321 08:57:48.715902 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:48.723863 28490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:48.724956 28489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:48.727319 28486 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35185
I20260321 08:57:48.727692 28486 heartbeater.cc:461] Registering TS with master...
W20260321 08:57:48.727944 28492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:48.728571 28486 heartbeater.cc:507] Master 127.24.0.190:35185 requested a full tablet report, sending...
I20260321 08:57:48.729054 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:48.729988 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:48.730185 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:48.730358 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083468730341 us; error 0 us; skew 500 ppm
I20260321 08:57:48.730492 28283 ts_manager.cc:194] Registered new tserver with Master: bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305)
I20260321 08:57:48.730847 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:48.732046 28283 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59104
I20260321 08:57:48.733456 24578 webserver.cc:492] Webserver started at http://127.24.0.131:41681/ using document root <none> and password file <none>
I20260321 08:57:48.734079 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:48.734334 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:48.734606 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:48.735679 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-2-root/instance:
uuid: "77e2de77a5b34546b569fc6b2faab03c"
format_stamp: "Formatted at 2026-03-21 08:57:48 on dist-test-slave-p54g"
I20260321 08:57:48.739789 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260321 08:57:48.742890 28497 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.743562 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:57:48.743818 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-2-root
uuid: "77e2de77a5b34546b569fc6b2faab03c"
format_stamp: "Formatted at 2026-03-21 08:57:48 on dist-test-slave-p54g"
I20260321 08:57:48.744117 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:48.760511 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:48.761556 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:48.762892 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:48.764983 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:48.765168 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.765385 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:48.765548 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:48.802763 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.131:36341
I20260321 08:57:48.802865 28559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.131:36341 every 8 connection(s)
I20260321 08:57:48.815858 28560 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35185
I20260321 08:57:48.816221 28560 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:48.816905 28560 heartbeater.cc:507] Master 127.24.0.190:35185 requested a full tablet report, sending...
I20260321 08:57:48.818533 28283 ts_manager.cc:194] Registered new tserver with Master: 77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341)
I20260321 08:57:48.819590 24578 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013644543s
I20260321 08:57:48.819769 28283 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59106
I20260321 08:57:48.860284 28283 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59122:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260321 08:57:48.901835 28525 tablet_service.cc:1511] Processing CreateTablet for tablet 35483005b1b74d188e7c01d3650988d2 (DEFAULT_TABLE table=test-workload [id=18c96368ae624987811f18c0c4f2c9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:48.903162 28525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35483005b1b74d188e7c01d3650988d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:48.902890 28376 tablet_service.cc:1511] Processing CreateTablet for tablet 35483005b1b74d188e7c01d3650988d2 (DEFAULT_TABLE table=test-workload [id=18c96368ae624987811f18c0c4f2c9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:48.904158 28376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35483005b1b74d188e7c01d3650988d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:48.918259 28580 tablet_bootstrap.cc:492] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Bootstrap starting.
I20260321 08:57:48.921263 28581 tablet_bootstrap.cc:492] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c: Bootstrap starting.
I20260321 08:57:48.922413 28451 tablet_service.cc:1511] Processing CreateTablet for tablet 35483005b1b74d188e7c01d3650988d2 (DEFAULT_TABLE table=test-workload [id=18c96368ae624987811f18c0c4f2c9b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:48.923753 28451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35483005b1b74d188e7c01d3650988d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:48.925396 28580 tablet_bootstrap.cc:654] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:48.927129 28581 tablet_bootstrap.cc:654] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:48.931036 28580 tablet_bootstrap.cc:492] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: No bootstrap required, opened a new log
I20260321 08:57:48.931622 28580 ts_tablet_manager.cc:1403] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260321 08:57:48.933882 28581 tablet_bootstrap.cc:492] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c: No bootstrap required, opened a new log
I20260321 08:57:48.934355 28581 ts_tablet_manager.cc:1403] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260321 08:57:48.934332 28580 raft_consensus.cc:359] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:48.935173 28580 raft_consensus.cc:385] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:48.935467 28580 raft_consensus.cc:740] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ed22d0c0af49e4a72844c10c323d55, State: Initialized, Role: FOLLOWER
I20260321 08:57:48.936190 28580 consensus_queue.cc:260] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:48.937263 28581 raft_consensus.cc:359] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:48.938024 28581 raft_consensus.cc:385] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:48.938325 28581 raft_consensus.cc:740] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e2de77a5b34546b569fc6b2faab03c, State: Initialized, Role: FOLLOWER
I20260321 08:57:48.939232 28581 consensus_queue.cc:260] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:48.942097 28585 tablet_bootstrap.cc:492] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01: Bootstrap starting.
I20260321 08:57:48.946707 28411 heartbeater.cc:499] Master 127.24.0.190:35185 was elected leader, sending a full tablet report...
I20260321 08:57:48.947510 28580 ts_tablet_manager.cc:1434] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260321 08:57:48.948362 28585 tablet_bootstrap.cc:654] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:48.950382 28581 ts_tablet_manager.cc:1434] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260321 08:57:48.951318 28560 heartbeater.cc:499] Master 127.24.0.190:35185 was elected leader, sending a full tablet report...
I20260321 08:57:48.955684 28585 tablet_bootstrap.cc:492] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01: No bootstrap required, opened a new log
I20260321 08:57:48.956091 28585 ts_tablet_manager.cc:1403] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260321 08:57:48.958983 28585 raft_consensus.cc:359] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:48.959705 28585 raft_consensus.cc:385] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:48.960021 28585 raft_consensus.cc:740] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4c7686acae461fb1d92c17ccb3ce01, State: Initialized, Role: FOLLOWER
I20260321 08:57:48.960703 28585 consensus_queue.cc:260] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:48.962795 28486 heartbeater.cc:499] Master 127.24.0.190:35185 was elected leader, sending a full tablet report...
I20260321 08:57:48.963200 28585 ts_tablet_manager.cc:1434] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260321 08:57:49.156895 28584 raft_consensus.cc:493] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:49.157363 28584 raft_consensus.cc:515] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:49.159102 28584 leader_election.cc:290] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341), bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305)
I20260321 08:57:49.169457 28535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35483005b1b74d188e7c01d3650988d2" candidate_uuid: "00ed22d0c0af49e4a72844c10c323d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e2de77a5b34546b569fc6b2faab03c" is_pre_election: true
I20260321 08:57:49.169853 28461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35483005b1b74d188e7c01d3650988d2" candidate_uuid: "00ed22d0c0af49e4a72844c10c323d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" is_pre_election: true
I20260321 08:57:49.170145 28535 raft_consensus.cc:2468] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00ed22d0c0af49e4a72844c10c323d55 in term 0.
I20260321 08:57:49.170473 28461 raft_consensus.cc:2468] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00ed22d0c0af49e4a72844c10c323d55 in term 0.
I20260321 08:57:49.171204 28350 leader_election.cc:304] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00ed22d0c0af49e4a72844c10c323d55, 77e2de77a5b34546b569fc6b2faab03c; no voters:
I20260321 08:57:49.171859 28584 raft_consensus.cc:2804] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:57:49.172163 28584 raft_consensus.cc:493] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:49.172381 28584 raft_consensus.cc:3060] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:49.177100 28584 raft_consensus.cc:515] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:49.178455 28584 leader_election.cc:290] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [CANDIDATE]: Term 1 election: Requested vote from peers 77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341), bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305)
I20260321 08:57:49.179245 28535 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35483005b1b74d188e7c01d3650988d2" candidate_uuid: "00ed22d0c0af49e4a72844c10c323d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e2de77a5b34546b569fc6b2faab03c"
I20260321 08:57:49.179438 28461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35483005b1b74d188e7c01d3650988d2" candidate_uuid: "00ed22d0c0af49e4a72844c10c323d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd4c7686acae461fb1d92c17ccb3ce01"
I20260321 08:57:49.179821 28535 raft_consensus.cc:3060] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:49.179963 28461 raft_consensus.cc:3060] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:49.186650 28535 raft_consensus.cc:2468] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00ed22d0c0af49e4a72844c10c323d55 in term 1.
I20260321 08:57:49.186645 28461 raft_consensus.cc:2468] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00ed22d0c0af49e4a72844c10c323d55 in term 1.
I20260321 08:57:49.188308 28349 leader_election.cc:304] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00ed22d0c0af49e4a72844c10c323d55, bd4c7686acae461fb1d92c17ccb3ce01; no voters:
I20260321 08:57:49.191139 28584 raft_consensus.cc:2804] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:49.193835 28584 raft_consensus.cc:697] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 1 LEADER]: Becoming Leader. State: Replica: 00ed22d0c0af49e4a72844c10c323d55, State: Running, Role: LEADER
I20260321 08:57:49.194592 28584 consensus_queue.cc:237] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:49.200397 28283 catalog_manager.cc:5671] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129). New cstate: current_term: 1 leader_uuid: "00ed22d0c0af49e4a72844c10c323d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } }
I20260321 08:57:49.505673 28376 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260321 08:57:49.507383 28525 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:49.507454 28451 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:49.608453 28584 consensus_queue.cc:1048] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:49.660351 28591 consensus_queue.cc:1048] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
68be4e29ce6b4b098455cd14a55203f5 | 127.24.0.190:35185 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
00ed22d0c0af49e4a72844c10c323d55 | 127.24.0.129:43537 | HEALTHY | <none> | 1 | 0
77e2de77a5b34546b569fc6b2faab03c | 127.24.0.131:36341 | HEALTHY | <none> | 0 | 0
bd4c7686acae461fb1d92c17ccb3ce01 | 127.24.0.130:33305 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 35483005b1b74d188e7c01d3650988d2 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341): RUNNING
00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): RUNNING [LEADER]
bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305): RUNNING
All reported replicas are:
A = 77e2de77a5b34546b569fc6b2faab03c
B = 00ed22d0c0af49e4a72844c10c323d55
C = bd4c7686acae461fb1d92c17ccb3ce01
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260321 08:57:49.730406 24578 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260321 08:57:50.075944 28376 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260321 08:57:50.085130 28525 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:50.086539 28451 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
68be4e29ce6b4b098455cd14a55203f5 | 127.24.0.190:35185 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
00ed22d0c0af49e4a72844c10c323d55 | 127.24.0.129:43537 | HEALTHY | <none> | 1 | 0
77e2de77a5b34546b569fc6b2faab03c | 127.24.0.131:36341 | HEALTHY | <none> | 0 | 0
bd4c7686acae461fb1d92c17ccb3ce01 | 127.24.0.130:33305 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260321 08:57:50.266777 24578 log_verifier.cc:126] Checking tablet 35483005b1b74d188e7c01d3650988d2
I20260321 08:57:50.322214 24578 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 35483005b1b74d188e7c01d3650988d2
I20260321 08:57:50.322764 24578 tablet_replica.cc:333] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: stopping tablet replica
I20260321 08:57:50.323240 24578 raft_consensus.cc:2243] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:50.324115 24578 raft_consensus.cc:2272] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:50.520443 28376 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:50.521797 28525 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:50.523729 28451 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
68be4e29ce6b4b098455cd14a55203f5 | 127.24.0.190:35185 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
00ed22d0c0af49e4a72844c10c323d55 | 127.24.0.129:43537 | HEALTHY | <none> | 0 | 0
77e2de77a5b34546b569fc6b2faab03c | 127.24.0.131:36341 | HEALTHY | <none> | 0 | 0
bd4c7686acae461fb1d92c17ccb3ce01 | 127.24.0.130:33305 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 35483005b1b74d188e7c01d3650988d2 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341): RUNNING
00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305): RUNNING
All reported replicas are:
A = 77e2de77a5b34546b569fc6b2faab03c
B = 00ed22d0c0af49e4a72844c10c323d55
C = bd4c7686acae461fb1d92c17ccb3ce01
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260321 08:57:50.719228 24578 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260321 08:57:51.073943 28451 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:51.074147 28525 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:51.074218 28376 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
68be4e29ce6b4b098455cd14a55203f5 | 127.24.0.190:35185 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
00ed22d0c0af49e4a72844c10c323d55 | 127.24.0.129:43537 | HEALTHY | <none> | 0 | 0
77e2de77a5b34546b569fc6b2faab03c | 127.24.0.131:36341 | HEALTHY | <none> | 0 | 0
bd4c7686acae461fb1d92c17ccb3ce01 | 127.24.0.130:33305 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 35483005b1b74d188e7c01d3650988d2 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341): RUNNING
00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305): RUNNING
All reported replicas are:
A = 77e2de77a5b34546b569fc6b2faab03c
B = 00ed22d0c0af49e4a72844c10c323d55
C = bd4c7686acae461fb1d92c17ccb3ce01
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260321 08:57:51.261624 24578 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260321 08:57:51.600035 28708 raft_consensus.cc:493] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00ed22d0c0af49e4a72844c10c323d55)
I20260321 08:57:51.600463 28708 raft_consensus.cc:515] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:51.602382 28708 leader_election.cc:290] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341), 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537)
I20260321 08:57:51.624135 28534 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35483005b1b74d188e7c01d3650988d2" candidate_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e2de77a5b34546b569fc6b2faab03c" is_pre_election: true
I20260321 08:57:51.627130 28386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35483005b1b74d188e7c01d3650988d2" candidate_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00ed22d0c0af49e4a72844c10c323d55" is_pre_election: true
W20260321 08:57:51.628140 28427 leader_election.cc:343] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): Illegal state: cannot vote while shut down
I20260321 08:57:51.628490 28427 leader_election.cc:304] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd4c7686acae461fb1d92c17ccb3ce01; no voters: 00ed22d0c0af49e4a72844c10c323d55, 77e2de77a5b34546b569fc6b2faab03c
I20260321 08:57:51.629102 28708 raft_consensus.cc:2749] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 08:57:51.634311 28718 raft_consensus.cc:493] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00ed22d0c0af49e4a72844c10c323d55)
I20260321 08:57:51.634706 28718 raft_consensus.cc:515] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:51.636627 28718 leader_election.cc:290] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537), bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305)
I20260321 08:57:51.650270 28386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35483005b1b74d188e7c01d3650988d2" candidate_uuid: "77e2de77a5b34546b569fc6b2faab03c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00ed22d0c0af49e4a72844c10c323d55" is_pre_election: true
W20260321 08:57:51.652014 28501 leader_election.cc:343] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): Illegal state: cannot vote while shut down
I20260321 08:57:51.660641 28461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35483005b1b74d188e7c01d3650988d2" candidate_uuid: "77e2de77a5b34546b569fc6b2faab03c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" is_pre_election: true
I20260321 08:57:51.661147 28461 raft_consensus.cc:2468] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e2de77a5b34546b569fc6b2faab03c in term 1.
I20260321 08:57:51.662103 28498 leader_election.cc:304] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e2de77a5b34546b569fc6b2faab03c, bd4c7686acae461fb1d92c17ccb3ce01; no voters: 00ed22d0c0af49e4a72844c10c323d55
I20260321 08:57:51.662746 28718 raft_consensus.cc:2804] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 08:57:51.663064 28718 raft_consensus.cc:493] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 00ed22d0c0af49e4a72844c10c323d55)
I20260321 08:57:51.663327 28718 raft_consensus.cc:3060] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:51.668221 28718 raft_consensus.cc:515] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:51.670565 28386 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35483005b1b74d188e7c01d3650988d2" candidate_uuid: "77e2de77a5b34546b569fc6b2faab03c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "00ed22d0c0af49e4a72844c10c323d55"
I20260321 08:57:51.670794 28718 leader_election.cc:290] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [CANDIDATE]: Term 2 election: Requested vote from peers 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537), bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305)
W20260321 08:57:51.672183 28501 leader_election.cc:343] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): Illegal state: cannot vote while shut down
I20260321 08:57:51.672477 28461 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35483005b1b74d188e7c01d3650988d2" candidate_uuid: "77e2de77a5b34546b569fc6b2faab03c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bd4c7686acae461fb1d92c17ccb3ce01"
I20260321 08:57:51.672888 28461 raft_consensus.cc:3060] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 1 FOLLOWER]: Advancing to term 2
I20260321 08:57:51.678194 28461 raft_consensus.cc:2468] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e2de77a5b34546b569fc6b2faab03c in term 2.
I20260321 08:57:51.678928 28498 leader_election.cc:304] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e2de77a5b34546b569fc6b2faab03c, bd4c7686acae461fb1d92c17ccb3ce01; no voters: 00ed22d0c0af49e4a72844c10c323d55
I20260321 08:57:51.679520 28718 raft_consensus.cc:2804] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 2 FOLLOWER]: Leader election won for term 2
I20260321 08:57:51.680514 28718 raft_consensus.cc:697] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 2 LEADER]: Becoming Leader. State: Replica: 77e2de77a5b34546b569fc6b2faab03c, State: Running, Role: LEADER
I20260321 08:57:51.681295 28718 consensus_queue.cc:237] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:51.687592 28281 catalog_manager.cc:5671] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c reported cstate change: term changed from 1 to 2, leader changed from 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129) to 77e2de77a5b34546b569fc6b2faab03c (127.24.0.131). New cstate: current_term: 2 leader_uuid: "77e2de77a5b34546b569fc6b2faab03c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } }
I20260321 08:57:51.717366 28376 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:51.719767 28525 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260321 08:57:51.722657 28451 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
68be4e29ce6b4b098455cd14a55203f5 | 127.24.0.190:35185 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
00ed22d0c0af49e4a72844c10c323d55 | 127.24.0.129:43537 | HEALTHY | <none> | 0 | 0
77e2de77a5b34546b569fc6b2faab03c | 127.24.0.131:36341 | HEALTHY | <none> | 1 | 0
bd4c7686acae461fb1d92c17ccb3ce01 | 127.24.0.130:33305 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 35483005b1b74d188e7c01d3650988d2 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341): RUNNING
00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305): RUNNING
All reported replicas are:
A = 77e2de77a5b34546b569fc6b2faab03c
B = 00ed22d0c0af49e4a72844c10c323d55
C = bd4c7686acae461fb1d92c17ccb3ce01
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260321 08:57:51.922271 24578 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260321 08:57:52.240876 28501 consensus_peers.cc:597] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c -> Peer 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): Couldn't send request to peer 00ed22d0c0af49e4a72844c10c323d55. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:52.256825 28461 raft_consensus.cc:1275] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 2 FOLLOWER]: Refusing update from remote peer 77e2de77a5b34546b569fc6b2faab03c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 08:57:52.258090 28739 consensus_queue.cc:1048] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 08:57:52.472281 28376 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:52.476044 28451 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:52.476171 28525 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
68be4e29ce6b4b098455cd14a55203f5 | 127.24.0.190:35185 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
00ed22d0c0af49e4a72844c10c323d55 | 127.24.0.129:43537 | HEALTHY | <none> | 0 | 0
77e2de77a5b34546b569fc6b2faab03c | 127.24.0.131:36341 | HEALTHY | <none> | 1 | 0
bd4c7686acae461fb1d92c17ccb3ce01 | 127.24.0.130:33305 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 35483005b1b74d188e7c01d3650988d2 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341): RUNNING [LEADER]
00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305): RUNNING
All reported replicas are:
A = 77e2de77a5b34546b569fc6b2faab03c
B = 00ed22d0c0af49e4a72844c10c323d55
C = bd4c7686acae461fb1d92c17ccb3ce01
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A* B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260321 08:57:52.668833 24578 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260321 08:57:52.690423 28739 raft_consensus.cc:1012] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c: Attempting to remove follower 00ed22d0c0af49e4a72844c10c323d55 from the Raft config. Reason: The tablet replica hosted on peer 00ed22d0c0af49e4a72844c10c323d55 has failed
I20260321 08:57:52.692493 28739 consensus_queue.cc:237] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } }
I20260321 08:57:52.695755 28461 raft_consensus.cc:1275] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 2 FOLLOWER]: Refusing update from remote peer 77e2de77a5b34546b569fc6b2faab03c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260321 08:57:52.696782 28772 consensus_queue.cc:1048] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 08:57:52.702124 28739 raft_consensus.cc:2955] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } }
I20260321 08:57:52.703454 28461 raft_consensus.cc:2955] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } }
I20260321 08:57:52.711107 28282 catalog_manager.cc:5671] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c reported cstate change: config changed from index -1 to 3, VOTER 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129) evicted. New cstate: current_term: 2 leader_uuid: "77e2de77a5b34546b569fc6b2faab03c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } }
I20260321 08:57:52.721465 28376 tablet_service.cc:1558] Processing DeleteTablet for tablet 35483005b1b74d188e7c01d3650988d2 with delete_type TABLET_DATA_TOMBSTONED (TS 00ed22d0c0af49e4a72844c10c323d55 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58920
I20260321 08:57:52.723553 28777 ts_tablet_manager.cc:1916] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 08:57:52.724397 28534 consensus_queue.cc:237] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } }
I20260321 08:57:52.728502 28461 raft_consensus.cc:1275] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 2 FOLLOWER]: Refusing update from remote peer 77e2de77a5b34546b569fc6b2faab03c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260321 08:57:52.729135 28501 consensus_peers.cc:597] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c -> Peer 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): Couldn't send request to peer 00ed22d0c0af49e4a72844c10c323d55. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:52.729648 28772 consensus_queue.cc:1048] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 08:57:52.734512 28739 raft_consensus.cc:2955] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } }
I20260321 08:57:52.735409 28777 ts_tablet_manager.cc:1929] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260321 08:57:52.735751 28777 log.cc:1199] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Deleting WAL directory at /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1774083407564530-24578-0/minicluster-data/ts-0-root/wals/35483005b1b74d188e7c01d3650988d2
I20260321 08:57:52.736069 28461 raft_consensus.cc:2955] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } }
I20260321 08:57:52.737002 28270 catalog_manager.cc:5002] TS 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): tablet 35483005b1b74d188e7c01d3650988d2 (table test-workload [id=18c96368ae624987811f18c0c4f2c9b4]) successfully deleted
I20260321 08:57:52.742743 28268 catalog_manager.cc:5184] ChangeConfig:ADD_PEER:VOTER RPC for tablet 35483005b1b74d188e7c01d3650988d2 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260321 08:57:52.745682 28282 catalog_manager.cc:5671] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c reported cstate change: config changed from index 3 to 4, VOTER 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129) added. New cstate: current_term: 2 leader_uuid: "77e2de77a5b34546b569fc6b2faab03c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } } }
I20260321 08:57:53.317709 28793 ts_tablet_manager.cc:933] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Initiating tablet copy from peer 77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341)
I20260321 08:57:53.319501 28793 tablet_copy_client.cc:323] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: tablet copy: Beginning tablet copy session from remote peer at address 127.24.0.131:36341
I20260321 08:57:53.320974 28545 tablet_copy_service.cc:140] P 77e2de77a5b34546b569fc6b2faab03c: Received BeginTabletCopySession request for tablet 35483005b1b74d188e7c01d3650988d2 from peer 00ed22d0c0af49e4a72844c10c323d55 ({username='slave'} at 127.0.0.1:36366)
I20260321 08:57:53.321468 28545 tablet_copy_service.cc:161] P 77e2de77a5b34546b569fc6b2faab03c: Beginning new tablet copy session on tablet 35483005b1b74d188e7c01d3650988d2 from peer 00ed22d0c0af49e4a72844c10c323d55 at {username='slave'} at 127.0.0.1:36366: session id = 00ed22d0c0af49e4a72844c10c323d55-35483005b1b74d188e7c01d3650988d2
I20260321 08:57:53.328181 28545 tablet_copy_source_session.cc:215] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 08:57:53.330307 28793 ts_tablet_manager.cc:1916] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260321 08:57:53.343705 28793 ts_tablet_manager.cc:1929] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260321 08:57:53.344267 28793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35483005b1b74d188e7c01d3650988d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:53.350231 28793 tablet_copy_client.cc:806] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: tablet copy: Starting download of 0 data blocks...
I20260321 08:57:53.350725 28793 tablet_copy_client.cc:670] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: tablet copy: Starting download of 1 WAL segments...
I20260321 08:57:53.353760 28793 tablet_copy_client.cc:538] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 08:57:53.360740 28793 tablet_bootstrap.cc:492] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Bootstrap starting.
I20260321 08:57:53.385185 28376 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:53.414695 28525 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260321 08:57:53.418001 28451 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:53.425907 28793 tablet_bootstrap.cc:492] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 08:57:53.426616 28793 tablet_bootstrap.cc:492] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Bootstrap complete.
I20260321 08:57:53.427143 28793 ts_tablet_manager.cc:1403] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Time spent bootstrapping tablet: real 0.067s user 0.026s sys 0.036s
I20260321 08:57:53.429296 28793 raft_consensus.cc:359] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } }
I20260321 08:57:53.429816 28793 raft_consensus.cc:740] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ed22d0c0af49e4a72844c10c323d55, State: Initialized, Role: FOLLOWER
I20260321 08:57:53.430436 28793 consensus_queue.cc:260] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e2de77a5b34546b569fc6b2faab03c" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 36341 } } peers { permanent_uuid: "bd4c7686acae461fb1d92c17ccb3ce01" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 33305 } } peers { permanent_uuid: "00ed22d0c0af49e4a72844c10c323d55" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 43537 } }
I20260321 08:57:53.432999 28793 ts_tablet_manager.cc:1434] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260321 08:57:53.434783 28545 tablet_copy_service.cc:342] P 77e2de77a5b34546b569fc6b2faab03c: Request end of tablet copy session 00ed22d0c0af49e4a72844c10c323d55-35483005b1b74d188e7c01d3650988d2 received from {username='slave'} at 127.0.0.1:36366
I20260321 08:57:53.435175 28545 tablet_copy_service.cc:434] P 77e2de77a5b34546b569fc6b2faab03c: ending tablet copy session 00ed22d0c0af49e4a72844c10c323d55-35483005b1b74d188e7c01d3650988d2 on tablet 35483005b1b74d188e7c01d3650988d2 with peer 00ed22d0c0af49e4a72844c10c323d55
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
68be4e29ce6b4b098455cd14a55203f5 | 127.24.0.190:35185 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
00ed22d0c0af49e4a72844c10c323d55 | 127.24.0.129:43537 | HEALTHY | <none> | 0 | 0
77e2de77a5b34546b569fc6b2faab03c | 127.24.0.131:36341 | HEALTHY | <none> | 1 | 0
bd4c7686acae461fb1d92c17ccb3ce01 | 127.24.0.130:33305 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 35483005b1b74d188e7c01d3650988d2 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
77e2de77a5b34546b569fc6b2faab03c (127.24.0.131:36341): RUNNING [LEADER]
bd4c7686acae461fb1d92c17ccb3ce01 (127.24.0.130:33305): RUNNING
00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = 77e2de77a5b34546b569fc6b2faab03c
B = bd4c7686acae461fb1d92c17ccb3ce01
C = 00ed22d0c0af49e4a72844c10c323d55
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | 4 | Yes
B | A* B C | 2 | 4 | Yes
C | A B C | 2 | 4 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260321 08:57:53.607321 24578 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260321 08:57:53.940032 28386 raft_consensus.cc:1217] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[]
I20260321 08:57:54.577765 28376 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:54.579088 28525 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260321 08:57:54.586594 28451 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
68be4e29ce6b4b098455cd14a55203f5 | 127.24.0.190:35185 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
00ed22d0c0af49e4a72844c10c323d55 | 127.24.0.129:43537 | HEALTHY | <none> | 0 | 0
77e2de77a5b34546b569fc6b2faab03c | 127.24.0.131:36341 | HEALTHY | <none> | 1 | 0
bd4c7686acae461fb1d92c17ccb3ce01 | 127.24.0.130:33305 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260321 08:57:54.786250 24578 log_verifier.cc:126] Checking tablet 35483005b1b74d188e7c01d3650988d2
I20260321 08:57:54.839152 24578 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 35483005b1b74d188e7c01d3650988d2
I20260321 08:57:54.842669 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:57:54.864235 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:54.864828 24578 tablet_replica.cc:333] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55: stopping tablet replica
I20260321 08:57:54.865301 24578 raft_consensus.cc:2243] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:54.865635 24578 raft_consensus.cc:2272] T 35483005b1b74d188e7c01d3650988d2 P 00ed22d0c0af49e4a72844c10c323d55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:54.884682 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:57:54.893646 24578 tablet_server.cc:179] TabletServer@127.24.0.130:0 shutting down...
I20260321 08:57:54.911584 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:54.912228 24578 tablet_replica.cc:333] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01: stopping tablet replica
I20260321 08:57:54.912729 24578 raft_consensus.cc:2243] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:54.913220 24578 raft_consensus.cc:2272] T 35483005b1b74d188e7c01d3650988d2 P bd4c7686acae461fb1d92c17ccb3ce01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:54.933410 24578 tablet_server.cc:196] TabletServer@127.24.0.130:0 shutdown complete.
W20260321 08:57:54.934594 28501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.0.129:43537: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20260321 08:57:54.937747 28501 consensus_peers.cc:597] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c -> Peer 00ed22d0c0af49e4a72844c10c323d55 (127.24.0.129:43537): Couldn't send request to peer 00ed22d0c0af49e4a72844c10c323d55. Status: Network error: Client connection negotiation failed: client connection to 127.24.0.129:43537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:54.943318 24578 tablet_server.cc:179] TabletServer@127.24.0.131:0 shutting down...
I20260321 08:57:54.959200 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:54.959793 24578 tablet_replica.cc:333] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c: stopping tablet replica
I20260321 08:57:54.960304 24578 raft_consensus.cc:2243] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 2 LEADER]: Raft consensus shutting down.
I20260321 08:57:54.961053 24578 raft_consensus.cc:2272] T 35483005b1b74d188e7c01d3650988d2 P 77e2de77a5b34546b569fc6b2faab03c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:54.979489 24578 tablet_server.cc:196] TabletServer@127.24.0.131:0 shutdown complete.
I20260321 08:57:54.988641 24578 master.cc:562] Master@127.24.0.190:35185 shutting down...
I20260321 08:57:55.005568 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:55.006071 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:55.006366 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 68be4e29ce6b4b098455cd14a55203f5: stopping tablet replica
I20260321 08:57:55.025520 24578 master.cc:584] Master@127.24.0.190:35185 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (6737 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20260321 08:57:55.053123 24578 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.24.0.190:35555
I20260321 08:57:55.054178 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:55.058650 28841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:55.060070 28842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:55.061602 28844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:55.062379 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:55.063319 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:55.063510 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:55.063657 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083475063643 us; error 0 us; skew 500 ppm
I20260321 08:57:55.064204 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:55.066529 24578 webserver.cc:492] Webserver started at http://127.24.0.190:44221/ using document root <none> and password file <none>
I20260321 08:57:55.066977 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:55.067137 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:55.067384 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:55.068434 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/master-0-root/instance:
uuid: "4b7e77f2f6c04fc4b24396d63083cf88"
format_stamp: "Formatted at 2026-03-21 08:57:55 on dist-test-slave-p54g"
I20260321 08:57:55.072592 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260321 08:57:55.075738 28849 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.076498 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:57:55.076767 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/master-0-root
uuid: "4b7e77f2f6c04fc4b24396d63083cf88"
format_stamp: "Formatted at 2026-03-21 08:57:55 on dist-test-slave-p54g"
I20260321 08:57:55.077011 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:55.087571 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:55.088686 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:55.125660 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.190:35555
I20260321 08:57:55.125743 28900 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.190:35555 every 8 connection(s)
I20260321 08:57:55.129415 28901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:55.139530 28901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88: Bootstrap starting.
I20260321 08:57:55.144622 28901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:55.148484 28901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88: No bootstrap required, opened a new log
I20260321 08:57:55.150350 28901 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e77f2f6c04fc4b24396d63083cf88" member_type: VOTER }
I20260321 08:57:55.150781 28901 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:55.150965 28901 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7e77f2f6c04fc4b24396d63083cf88, State: Initialized, Role: FOLLOWER
I20260321 08:57:55.151470 28901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e77f2f6c04fc4b24396d63083cf88" member_type: VOTER }
I20260321 08:57:55.151921 28901 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 08:57:55.152125 28901 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 08:57:55.152355 28901 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:55.157059 28901 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e77f2f6c04fc4b24396d63083cf88" member_type: VOTER }
I20260321 08:57:55.157531 28901 leader_election.cc:304] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7e77f2f6c04fc4b24396d63083cf88; no voters:
I20260321 08:57:55.158530 28901 leader_election.cc:290] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 08:57:55.158918 28904 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:55.160296 28904 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 1 LEADER]: Becoming Leader. State: Replica: 4b7e77f2f6c04fc4b24396d63083cf88, State: Running, Role: LEADER
I20260321 08:57:55.160861 28904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e77f2f6c04fc4b24396d63083cf88" member_type: VOTER }
I20260321 08:57:55.161356 28901 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 08:57:55.163460 28905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b7e77f2f6c04fc4b24396d63083cf88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e77f2f6c04fc4b24396d63083cf88" member_type: VOTER } }
I20260321 08:57:55.163581 28906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7e77f2f6c04fc4b24396d63083cf88. Latest consensus state: current_term: 1 leader_uuid: "4b7e77f2f6c04fc4b24396d63083cf88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7e77f2f6c04fc4b24396d63083cf88" member_type: VOTER } }
I20260321 08:57:55.164055 28905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:55.164301 28906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [sys.catalog]: This master's current role is: LEADER
I20260321 08:57:55.166776 28909 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 08:57:55.170954 28909 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 08:57:55.175127 24578 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 08:57:55.179168 28909 catalog_manager.cc:1357] Generated new cluster ID: a6c7d438a2f7479e8083d710b307072e
I20260321 08:57:55.179435 28909 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 08:57:55.204473 28909 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 08:57:55.205729 28909 catalog_manager.cc:1514] Loading token signing keys...
I20260321 08:57:55.217449 28909 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88: Generated new TSK 0
I20260321 08:57:55.218065 28909 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 08:57:55.241138 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:55.246598 28922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:55.247427 28923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:55.249454 28925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:55.250492 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:55.252772 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:55.252960 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:55.253119 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083475253105 us; error 0 us; skew 500 ppm
I20260321 08:57:55.253618 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:55.255735 24578 webserver.cc:492] Webserver started at http://127.24.0.129:42997/ using document root <none> and password file <none>
I20260321 08:57:55.256191 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:55.256376 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:55.256614 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:55.257601 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-0-root/instance:
uuid: "c6d785bc765f4fd89929f4134ca5226a"
format_stamp: "Formatted at 2026-03-21 08:57:55 on dist-test-slave-p54g"
I20260321 08:57:55.261790 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260321 08:57:55.265123 28930 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.265817 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260321 08:57:55.266088 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-0-root
uuid: "c6d785bc765f4fd89929f4134ca5226a"
format_stamp: "Formatted at 2026-03-21 08:57:55 on dist-test-slave-p54g"
I20260321 08:57:55.266362 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:55.279827 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:55.280890 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:55.282159 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:55.284209 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:55.284405 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.284636 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:55.284785 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.320050 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.129:40933
I20260321 08:57:55.320152 28992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.129:40933 every 8 connection(s)
I20260321 08:57:55.324488 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:55.330281 28997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:55.330855 28998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:55.333776 29000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:55.334354 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:55.335335 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:55.335533 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:55.335728 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083475335709 us; error 0 us; skew 500 ppm
I20260321 08:57:55.335778 28993 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35555
I20260321 08:57:55.336194 28993 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:55.336364 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:55.336856 28993 heartbeater.cc:507] Master 127.24.0.190:35555 requested a full tablet report, sending...
I20260321 08:57:55.338861 24578 webserver.cc:492] Webserver started at http://127.24.0.130:36307/ using document root <none> and password file <none>
I20260321 08:57:55.338917 28866 ts_manager.cc:194] Registered new tserver with Master: c6d785bc765f4fd89929f4134ca5226a (127.24.0.129:40933)
I20260321 08:57:55.339421 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:55.339624 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:55.339850 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:55.340723 28866 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58600
I20260321 08:57:55.340876 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-1-root/instance:
uuid: "71c8a91e37bd45249a30d9340b095b3b"
format_stamp: "Formatted at 2026-03-21 08:57:55 on dist-test-slave-p54g"
I20260321 08:57:55.345449 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260321 08:57:55.348395 29005 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.349104 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260321 08:57:55.349397 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-1-root
uuid: "71c8a91e37bd45249a30d9340b095b3b"
format_stamp: "Formatted at 2026-03-21 08:57:55 on dist-test-slave-p54g"
I20260321 08:57:55.349689 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:55.367491 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:55.368541 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:55.369833 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:55.372006 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:55.372197 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.372426 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:55.372578 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.409195 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.130:44989
I20260321 08:57:55.409307 29067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.130:44989 every 8 connection(s)
I20260321 08:57:55.413738 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:55.421510 29071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:55.422844 29072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:55.425011 29068 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35555
I20260321 08:57:55.425460 29068 heartbeater.cc:461] Registering TS with master...
W20260321 08:57:55.425623 29074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:55.426553 29068 heartbeater.cc:507] Master 127.24.0.190:35555 requested a full tablet report, sending...
I20260321 08:57:55.427001 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:55.427838 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:55.428118 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:55.428319 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083475428300 us; error 0 us; skew 500 ppm
I20260321 08:57:55.428805 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:55.428766 28866 ts_manager.cc:194] Registered new tserver with Master: 71c8a91e37bd45249a30d9340b095b3b (127.24.0.130:44989)
I20260321 08:57:55.430436 28866 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58616
I20260321 08:57:55.431384 24578 webserver.cc:492] Webserver started at http://127.24.0.131:43857/ using document root <none> and password file <none>
I20260321 08:57:55.431911 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:55.432098 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:55.432344 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:55.433360 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-2-root/instance:
uuid: "b24da5c14c8e427ea8c1a5d44f1ba259"
format_stamp: "Formatted at 2026-03-21 08:57:55 on dist-test-slave-p54g"
I20260321 08:57:55.437579 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260321 08:57:55.440862 29079 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.441658 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260321 08:57:55.441910 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-2-root
uuid: "b24da5c14c8e427ea8c1a5d44f1ba259"
format_stamp: "Formatted at 2026-03-21 08:57:55 on dist-test-slave-p54g"
I20260321 08:57:55.442143 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:55.460238 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:55.461254 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:55.462535 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:55.464787 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:55.464964 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.465193 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:55.465345 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.502066 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.131:38869
I20260321 08:57:55.502192 29141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.131:38869 every 8 connection(s)
I20260321 08:57:55.506256 24578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 08:57:55.513715 29145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 08:57:55.513814 29146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:55.516585 29142 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35555
W20260321 08:57:55.516901 29148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 08:57:55.517066 29142 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:55.517804 24578 server_base.cc:1061] running on GCE node
I20260321 08:57:55.518046 29142 heartbeater.cc:507] Master 127.24.0.190:35555 requested a full tablet report, sending...
I20260321 08:57:55.518777 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 08:57:55.519025 24578 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 08:57:55.519209 24578 hybrid_clock.cc:648] HybridClock initialized: now 1774083475519192 us; error 0 us; skew 500 ppm
I20260321 08:57:55.519858 24578 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 08:57:55.520150 28866 ts_manager.cc:194] Registered new tserver with Master: b24da5c14c8e427ea8c1a5d44f1ba259 (127.24.0.131:38869)
I20260321 08:57:55.521695 28866 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58620
I20260321 08:57:55.522269 24578 webserver.cc:492] Webserver started at http://127.24.0.132:40999/ using document root <none> and password file <none>
I20260321 08:57:55.522866 24578 fs_manager.cc:362] Metadata directory not provided
I20260321 08:57:55.523037 24578 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 08:57:55.523269 24578 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 08:57:55.524328 24578 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-3-root/instance:
uuid: "b1de9b7e1f67446e962cc8ed175c1196"
format_stamp: "Formatted at 2026-03-21 08:57:55 on dist-test-slave-p54g"
I20260321 08:57:55.528529 24578 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260321 08:57:55.531623 29153 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.532368 24578 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260321 08:57:55.532640 24578 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-3-root
uuid: "b1de9b7e1f67446e962cc8ed175c1196"
format_stamp: "Formatted at 2026-03-21 08:57:55 on dist-test-slave-p54g"
I20260321 08:57:55.532920 24578 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 08:57:55.553733 24578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 08:57:55.554776 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 08:57:55.556094 24578 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 08:57:55.558146 24578 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 08:57:55.558348 24578 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.558573 24578 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 08:57:55.558722 24578 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 08:57:55.594906 24578 rpc_server.cc:307] RPC server started. Bound to: 127.24.0.132:46225
I20260321 08:57:55.595006 29215 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.24.0.132:46225 every 8 connection(s)
I20260321 08:57:55.608029 29216 heartbeater.cc:344] Connected to a master server at 127.24.0.190:35555
I20260321 08:57:55.608428 29216 heartbeater.cc:461] Registering TS with master...
I20260321 08:57:55.609113 29216 heartbeater.cc:507] Master 127.24.0.190:35555 requested a full tablet report, sending...
I20260321 08:57:55.610917 28866 ts_manager.cc:194] Registered new tserver with Master: b1de9b7e1f67446e962cc8ed175c1196 (127.24.0.132:46225)
I20260321 08:57:55.611395 24578 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.012469055s
I20260321 08:57:55.612558 28866 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58624
I20260321 08:57:55.649034 28866 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58648:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260321 08:57:55.690986 28958 tablet_service.cc:1511] Processing CreateTablet for tablet 0fbc8569512b4954b8a01aa8491fa27e (DEFAULT_TABLE table=test-workload [id=9d5c6e8d9211406e80014548309f8256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:55.691098 29107 tablet_service.cc:1511] Processing CreateTablet for tablet 0fbc8569512b4954b8a01aa8491fa27e (DEFAULT_TABLE table=test-workload [id=9d5c6e8d9211406e80014548309f8256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:55.692188 29107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fbc8569512b4954b8a01aa8491fa27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:55.692200 28958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fbc8569512b4954b8a01aa8491fa27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:55.697788 29033 tablet_service.cc:1511] Processing CreateTablet for tablet 0fbc8569512b4954b8a01aa8491fa27e (DEFAULT_TABLE table=test-workload [id=9d5c6e8d9211406e80014548309f8256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260321 08:57:55.698729 29033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fbc8569512b4954b8a01aa8491fa27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 08:57:55.708055 29236 tablet_bootstrap.cc:492] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259: Bootstrap starting.
I20260321 08:57:55.709163 29237 tablet_bootstrap.cc:492] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a: Bootstrap starting.
I20260321 08:57:55.714421 29237 tablet_bootstrap.cc:654] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:55.715085 29236 tablet_bootstrap.cc:654] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:55.715627 29238 tablet_bootstrap.cc:492] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b: Bootstrap starting.
I20260321 08:57:55.719415 29237 tablet_bootstrap.cc:492] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a: No bootstrap required, opened a new log
I20260321 08:57:55.719853 29236 tablet_bootstrap.cc:492] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259: No bootstrap required, opened a new log
I20260321 08:57:55.719866 29237 ts_tablet_manager.cc:1403] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260321 08:57:55.720510 29236 ts_tablet_manager.cc:1403] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260321 08:57:55.722210 29238 tablet_bootstrap.cc:654] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b: Neither blocks nor log segments found. Creating new log.
I20260321 08:57:55.723062 29237 raft_consensus.cc:359] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 } } peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } }
I20260321 08:57:55.723258 29236 raft_consensus.cc:359] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 } } peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } }
I20260321 08:57:55.723866 29237 raft_consensus.cc:385] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:55.724016 29236 raft_consensus.cc:385] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:55.724231 29237 raft_consensus.cc:740] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d785bc765f4fd89929f4134ca5226a, State: Initialized, Role: FOLLOWER
I20260321 08:57:55.724352 29236 raft_consensus.cc:740] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24da5c14c8e427ea8c1a5d44f1ba259, State: Initialized, Role: FOLLOWER
I20260321 08:57:55.724995 29237 consensus_queue.cc:260] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 } } peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } }
I20260321 08:57:55.725041 29236 consensus_queue.cc:260] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 } } peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } }
I20260321 08:57:55.727319 28993 heartbeater.cc:499] Master 127.24.0.190:35555 was elected leader, sending a full tablet report...
I20260321 08:57:55.728297 29237 ts_tablet_manager.cc:1434] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260321 08:57:55.728960 29142 heartbeater.cc:499] Master 127.24.0.190:35555 was elected leader, sending a full tablet report...
I20260321 08:57:55.730046 29236 ts_tablet_manager.cc:1434] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20260321 08:57:55.732760 29238 tablet_bootstrap.cc:492] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b: No bootstrap required, opened a new log
I20260321 08:57:55.733256 29238 ts_tablet_manager.cc:1403] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20260321 08:57:55.736073 29238 raft_consensus.cc:359] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 } } peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } }
I20260321 08:57:55.736757 29238 raft_consensus.cc:385] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 08:57:55.737038 29238 raft_consensus.cc:740] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c8a91e37bd45249a30d9340b095b3b, State: Initialized, Role: FOLLOWER
I20260321 08:57:55.737733 29238 consensus_queue.cc:260] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 } } peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } }
I20260321 08:57:55.740212 29068 heartbeater.cc:499] Master 127.24.0.190:35555 was elected leader, sending a full tablet report...
I20260321 08:57:55.740752 29238 ts_tablet_manager.cc:1434] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260321 08:57:55.757850 29243 raft_consensus.cc:493] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 08:57:55.758294 29243 raft_consensus.cc:515] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 } } peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } }
I20260321 08:57:55.760145 29243 leader_election.cc:290] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6d785bc765f4fd89929f4134ca5226a (127.24.0.129:40933), 71c8a91e37bd45249a30d9340b095b3b (127.24.0.130:44989)
I20260321 08:57:55.769474 28968 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fbc8569512b4954b8a01aa8491fa27e" candidate_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d785bc765f4fd89929f4134ca5226a" is_pre_election: true
I20260321 08:57:55.770076 28968 raft_consensus.cc:2468] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24da5c14c8e427ea8c1a5d44f1ba259 in term 0.
I20260321 08:57:55.771010 29080 leader_election.cc:304] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24da5c14c8e427ea8c1a5d44f1ba259, c6d785bc765f4fd89929f4134ca5226a; no voters:
I20260321 08:57:55.771071 29043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fbc8569512b4954b8a01aa8491fa27e" candidate_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c8a91e37bd45249a30d9340b095b3b" is_pre_election: true
I20260321 08:57:55.771808 29043 raft_consensus.cc:2468] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24da5c14c8e427ea8c1a5d44f1ba259 in term 0.
I20260321 08:57:55.771867 29243 raft_consensus.cc:2804] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 08:57:55.772230 29243 raft_consensus.cc:493] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 08:57:55.772528 29243 raft_consensus.cc:3060] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:55.776949 29243 raft_consensus.cc:515] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 } } peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } }
I20260321 08:57:55.778288 29243 leader_election.cc:290] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [CANDIDATE]: Term 1 election: Requested vote from peers c6d785bc765f4fd89929f4134ca5226a (127.24.0.129:40933), 71c8a91e37bd45249a30d9340b095b3b (127.24.0.130:44989)
I20260321 08:57:55.779028 28968 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fbc8569512b4954b8a01aa8491fa27e" candidate_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6d785bc765f4fd89929f4134ca5226a"
I20260321 08:57:55.779134 29043 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fbc8569512b4954b8a01aa8491fa27e" candidate_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71c8a91e37bd45249a30d9340b095b3b"
I20260321 08:57:55.779450 28968 raft_consensus.cc:3060] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:55.779592 29043 raft_consensus.cc:3060] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 0 FOLLOWER]: Advancing to term 1
I20260321 08:57:55.783782 28968 raft_consensus.cc:2468] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24da5c14c8e427ea8c1a5d44f1ba259 in term 1.
I20260321 08:57:55.783864 29043 raft_consensus.cc:2468] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24da5c14c8e427ea8c1a5d44f1ba259 in term 1.
I20260321 08:57:55.784670 29080 leader_election.cc:304] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b24da5c14c8e427ea8c1a5d44f1ba259, c6d785bc765f4fd89929f4134ca5226a; no voters:
I20260321 08:57:55.785424 29243 raft_consensus.cc:2804] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 08:57:55.786580 29243 raft_consensus.cc:697] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 1 LEADER]: Becoming Leader. State: Replica: b24da5c14c8e427ea8c1a5d44f1ba259, State: Running, Role: LEADER
I20260321 08:57:55.787261 29243 consensus_queue.cc:237] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 } } peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } }
I20260321 08:57:55.793330 28865 catalog_manager.cc:5671] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 reported cstate change: term changed from 0 to 1, leader changed from <none> to b24da5c14c8e427ea8c1a5d44f1ba259 (127.24.0.131). New cstate: current_term: 1 leader_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } health_report { overall_health: UNKNOWN } } }
I20260321 08:57:56.028535 29033 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:56.045037 29181 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:56.045224 29107 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260321 08:57:56.051421 28958 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
4b7e77f2f6c04fc4b24396d63083cf88 | 127.24.0.190:35555 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
71c8a91e37bd45249a30d9340b095b3b | 127.24.0.130:44989 | HEALTHY | <none> | 0 | 0
b1de9b7e1f67446e962cc8ed175c1196 | 127.24.0.132:46225 | HEALTHY | <none> | 0 | 0
b24da5c14c8e427ea8c1a5d44f1ba259 | 127.24.0.131:38869 | HEALTHY | <none> | 1 | 0
c6d785bc765f4fd89929f4134ca5226a | 127.24.0.129:40933 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 0fbc8569512b4954b8a01aa8491fa27e of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
c6d785bc765f4fd89929f4134ca5226a (127.24.0.129:40933): RUNNING
b24da5c14c8e427ea8c1a5d44f1ba259 (127.24.0.131:38869): RUNNING [LEADER]
71c8a91e37bd45249a30d9340b095b3b (127.24.0.130:44989): RUNNING
All reported replicas are:
A = c6d785bc765f4fd89929f4134ca5226a
B = b24da5c14c8e427ea8c1a5d44f1ba259
C = 71c8a91e37bd45249a30d9340b095b3b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | b1de9b7e1f67446e962cc8ed175c1196 | 127.24.0.132:46225 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260321 08:57:56.291170 24578 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260321 08:57:56.380492 29277 consensus_queue.cc:1048] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:56.393641 29277 consensus_queue.cc:1048] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d785bc765f4fd89929f4134ca5226a" member_type: VOTER last_known_addr { host: "127.24.0.129" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 08:57:56.615123 29216 heartbeater.cc:499] Master 127.24.0.190:35555 was elected leader, sending a full tablet report...
I20260321 08:57:56.639047 29033 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:56.645869 29181 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:56.650298 29107 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260321 08:57:56.651352 28958 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
4b7e77f2f6c04fc4b24396d63083cf88 | 127.24.0.190:35555 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
71c8a91e37bd45249a30d9340b095b3b | 127.24.0.130:44989 | HEALTHY | <none> | 0 | 0
b1de9b7e1f67446e962cc8ed175c1196 | 127.24.0.132:46225 | HEALTHY | <none> | 0 | 0
b24da5c14c8e427ea8c1a5d44f1ba259 | 127.24.0.131:38869 | HEALTHY | <none> | 1 | 0
c6d785bc765f4fd89929f4134ca5226a | 127.24.0.129:40933 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | b1de9b7e1f67446e962cc8ed175c1196 | 127.24.0.132:46225 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260321 08:57:56.880419 24578 log_verifier.cc:126] Checking tablet 0fbc8569512b4954b8a01aa8491fa27e
I20260321 08:57:56.933622 24578 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0fbc8569512b4954b8a01aa8491fa27e
I20260321 08:57:56.934130 24578 tablet_replica.cc:333] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a: stopping tablet replica
I20260321 08:57:56.934629 24578 raft_consensus.cc:2243] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:56.935088 24578 raft_consensus.cc:2272] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:56.954157 29080 consensus_queue.cc:579] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [LEADER]: The tablet replica hosted on peer c6d785bc765f4fd89929f4134ca5226a has failed
W20260321 08:57:56.954608 29080 consensus_peers.cc:597] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 -> Peer c6d785bc765f4fd89929f4134ca5226a (127.24.0.129:40933): Couldn't send request to peer c6d785bc765f4fd89929f4134ca5226a. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:56.966485 29117 consensus_queue.cc:237] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } }
I20260321 08:57:56.970782 29043 raft_consensus.cc:1275] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 1 FOLLOWER]: Refusing update from remote peer b24da5c14c8e427ea8c1a5d44f1ba259: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 08:57:56.971977 29318 consensus_queue.cc:1048] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 08:57:56.977161 29322 raft_consensus.cc:2955] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c6d785bc765f4fd89929f4134ca5226a (127.24.0.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } } }
I20260321 08:57:56.978665 29043 raft_consensus.cc:2955] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER c6d785bc765f4fd89929f4134ca5226a (127.24.0.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } } }
I20260321 08:57:56.986475 28851 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 0fbc8569512b4954b8a01aa8491fa27e with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 08:57:56.991138 28866 catalog_manager.cc:5671] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 reported cstate change: config changed from index -1 to 2, VOTER c6d785bc765f4fd89929f4134ca5226a (127.24.0.129) evicted. New cstate: current_term: 1 leader_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } health_report { overall_health: HEALTHY } } }
I20260321 08:57:57.004570 28958 tablet_service.cc:1558] Processing DeleteTablet for tablet 0fbc8569512b4954b8a01aa8491fa27e with delete_type TABLET_DATA_TOMBSTONED (TS c6d785bc765f4fd89929f4134ca5226a not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:39564
I20260321 08:57:57.006397 29329 ts_tablet_manager.cc:1916] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 08:57:57.008126 29117 consensus_queue.cc:237] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } } peers { permanent_uuid: "b1de9b7e1f67446e962cc8ed175c1196" member_type: NON_VOTER last_known_addr { host: "127.24.0.132" port: 46225 } attrs { promote: true } }
I20260321 08:57:57.017405 29043 raft_consensus.cc:1275] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 1 FOLLOWER]: Refusing update from remote peer b24da5c14c8e427ea8c1a5d44f1ba259: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 08:57:57.019018 29322 consensus_queue.cc:1048] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 08:57:57.022681 29329 ts_tablet_manager.cc:1929] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260321 08:57:57.023149 29329 log.cc:1199] T 0fbc8569512b4954b8a01aa8491fa27e P c6d785bc765f4fd89929f4134ca5226a: Deleting WAL directory at /tmp/dist-test-task96D6Sv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1774083407564530-24578-0/minicluster-data/ts-0-root/wals/0fbc8569512b4954b8a01aa8491fa27e
I20260321 08:57:57.024925 28850 catalog_manager.cc:5002] TS c6d785bc765f4fd89929f4134ca5226a (127.24.0.129:40933): tablet 0fbc8569512b4954b8a01aa8491fa27e (table test-workload [id=9d5c6e8d9211406e80014548309f8256]) successfully deleted
I20260321 08:57:57.026662 29322 raft_consensus.cc:2955] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b1de9b7e1f67446e962cc8ed175c1196 (127.24.0.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } } peers { permanent_uuid: "b1de9b7e1f67446e962cc8ed175c1196" member_type: NON_VOTER last_known_addr { host: "127.24.0.132" port: 46225 } attrs { promote: true } } }
I20260321 08:57:57.028348 29043 raft_consensus.cc:2955] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b1de9b7e1f67446e962cc8ed175c1196 (127.24.0.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } } peers { permanent_uuid: "b1de9b7e1f67446e962cc8ed175c1196" member_type: NON_VOTER last_known_addr { host: "127.24.0.132" port: 46225 } attrs { promote: true } } }
I20260321 08:57:57.038616 28851 catalog_manager.cc:5184] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0fbc8569512b4954b8a01aa8491fa27e with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260321 08:57:57.041425 28866 catalog_manager.cc:5671] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b reported cstate change: config changed from index 2 to 3, NON_VOTER b1de9b7e1f67446e962cc8ed175c1196 (127.24.0.132) added. New cstate: current_term: 1 leader_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b24da5c14c8e427ea8c1a5d44f1ba259" member_type: VOTER last_known_addr { host: "127.24.0.131" port: 38869 } } peers { permanent_uuid: "71c8a91e37bd45249a30d9340b095b3b" member_type: VOTER last_known_addr { host: "127.24.0.130" port: 44989 } } peers { permanent_uuid: "b1de9b7e1f67446e962cc8ed175c1196" member_type: NON_VOTER last_known_addr { host: "127.24.0.132" port: 46225 } attrs { promote: true } } }
W20260321 08:57:57.045006 29080 consensus_peers.cc:597] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 -> Peer b1de9b7e1f67446e962cc8ed175c1196 (127.24.0.132:46225): Couldn't send request to peer b1de9b7e1f67446e962cc8ed175c1196. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fbc8569512b4954b8a01aa8491fa27e. This is attempt 1: this message will repeat every 5th retry.
I20260321 08:57:57.170583 29033 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:57.171370 29181 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260321 08:57:57.180620 29107 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260321 08:57:57.182701 28958 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
4b7e77f2f6c04fc4b24396d63083cf88 | 127.24.0.190:35555 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
71c8a91e37bd45249a30d9340b095b3b | 127.24.0.130:44989 | HEALTHY | <none> | 0 | 0
b1de9b7e1f67446e962cc8ed175c1196 | 127.24.0.132:46225 | HEALTHY | <none> | 0 | 0
b24da5c14c8e427ea8c1a5d44f1ba259 | 127.24.0.131:38869 | HEALTHY | <none> | 1 | 0
c6d785bc765f4fd89929f4134ca5226a | 127.24.0.129:40933 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+--------------------+---------------
Small | c6d785bc765f4fd89929f4134ca5226a | 127.24.0.129:40933 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260321 08:57:57.396625 24578 log_verifier.cc:126] Checking tablet 0fbc8569512b4954b8a01aa8491fa27e
I20260321 08:57:57.430013 24578 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0fbc8569512b4954b8a01aa8491fa27e
I20260321 08:57:57.431842 24578 tablet_server.cc:179] TabletServer@127.24.0.129:0 shutting down...
I20260321 08:57:57.450832 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:57.467479 24578 tablet_server.cc:196] TabletServer@127.24.0.129:0 shutdown complete.
I20260321 08:57:57.476284 24578 tablet_server.cc:179] TabletServer@127.24.0.130:0 shutting down...
I20260321 08:57:57.493044 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:57.493628 24578 tablet_replica.cc:333] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b: stopping tablet replica
I20260321 08:57:57.494108 24578 raft_consensus.cc:2243] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 08:57:57.494563 24578 raft_consensus.cc:2272] T 0fbc8569512b4954b8a01aa8491fa27e P 71c8a91e37bd45249a30d9340b095b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:57.513278 24578 tablet_server.cc:196] TabletServer@127.24.0.130:0 shutdown complete.
I20260321 08:57:57.521467 24578 tablet_server.cc:179] TabletServer@127.24.0.131:0 shutting down...
I20260321 08:57:57.537788 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:57.538379 24578 tablet_replica.cc:333] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259: stopping tablet replica
I20260321 08:57:57.538856 24578 raft_consensus.cc:2243] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:57.539777 24578 raft_consensus.cc:2272] T 0fbc8569512b4954b8a01aa8491fa27e P b24da5c14c8e427ea8c1a5d44f1ba259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:57.558394 24578 tablet_server.cc:196] TabletServer@127.24.0.131:0 shutdown complete.
I20260321 08:57:57.566323 24578 tablet_server.cc:179] TabletServer@127.24.0.132:0 shutting down...
I20260321 08:57:57.582612 24578 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 08:57:57.597378 24578 tablet_server.cc:196] TabletServer@127.24.0.132:0 shutdown complete.
I20260321 08:57:57.605363 24578 master.cc:562] Master@127.24.0.190:35555 shutting down...
I20260321 08:57:57.619540 24578 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 1 LEADER]: Raft consensus shutting down.
I20260321 08:57:57.620059 24578 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 08:57:57.620361 24578 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4b7e77f2f6c04fc4b24396d63083cf88: stopping tablet replica
I20260321 08:57:57.638232 24578 master.cc:584] Master@127.24.0.190:35555 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (2609 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (9346 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (69935 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20260321 08:57:57.669418 24578 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~10 seconds ago
I20260321 08:57:57.669701 24578 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 218 messages since previous log ~61 seconds ago
I20260321 08:57:57.669986 24578 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2379: suppressed but not reported on 7 messages since previous log ~13 seconds ago