[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260129 03:00:00.387970 23477 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.075s user 0.407s sys 0.666s
W20260129 03:00:00.388643 23477 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.076s user 0.407s sys 0.666s
I20260129 03:00:00.391633 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:41363
I20260129 03:00:00.394007 23477 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260129 03:00:00.395697 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:00.411206 23488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:00.412518 23489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:00.540609 23491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:00.540848 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:00.541992 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:00.542210 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:00.542358 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655600542342 us; error 0 us; skew 500 ppm
I20260129 03:00:00.543037 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:00.547739 23477 webserver.cc:492] Webserver started at http://127.22.237.126:37299/ using document root <none> and password file <none>
I20260129 03:00:00.548519 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:00.548715 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:00.549081 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:00.554780 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "078fe9f1cd8d41f8838f304177e649c5"
format_stamp: "Formatted at 2026-01-29 03:00:00 on dist-test-slave-1zcf"
I20260129 03:00:00.562510 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260129 03:00:00.568387 23496 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:00.569538 23477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260129 03:00:00.569900 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "078fe9f1cd8d41f8838f304177e649c5"
format_stamp: "Formatted at 2026-01-29 03:00:00 on dist-test-slave-1zcf"
I20260129 03:00:00.570266 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:00.606361 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:00.607836 23477 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260129 03:00:00.608296 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:00.677827 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:41363
I20260129 03:00:00.677881 23547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:41363 every 8 connection(s)
I20260129 03:00:00.684159 23548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:00.706198 23548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5: Bootstrap starting.
I20260129 03:00:00.712419 23548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:00.714262 23548 log.cc:826] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5: Log is configured to *not* fsync() on all Append() calls
I20260129 03:00:00.718885 23548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5: No bootstrap required, opened a new log
I20260129 03:00:00.739181 23548 raft_consensus.cc:359] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078fe9f1cd8d41f8838f304177e649c5" member_type: VOTER }
I20260129 03:00:00.739751 23548 raft_consensus.cc:385] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:00.739953 23548 raft_consensus.cc:740] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078fe9f1cd8d41f8838f304177e649c5, State: Initialized, Role: FOLLOWER
I20260129 03:00:00.740672 23548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078fe9f1cd8d41f8838f304177e649c5" member_type: VOTER }
I20260129 03:00:00.741139 23548 raft_consensus.cc:399] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:00.741380 23548 raft_consensus.cc:493] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:00.741683 23548 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:00.745752 23548 raft_consensus.cc:515] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078fe9f1cd8d41f8838f304177e649c5" member_type: VOTER }
I20260129 03:00:00.746455 23548 leader_election.cc:304] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 078fe9f1cd8d41f8838f304177e649c5; no voters:
I20260129 03:00:00.748106 23548 leader_election.cc:290] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:00.748483 23551 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:00.750556 23551 raft_consensus.cc:697] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 1 LEADER]: Becoming Leader. State: Replica: 078fe9f1cd8d41f8838f304177e649c5, State: Running, Role: LEADER
I20260129 03:00:00.751358 23551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078fe9f1cd8d41f8838f304177e649c5" member_type: VOTER }
I20260129 03:00:00.752434 23548 sys_catalog.cc:565] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:00.759017 23552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "078fe9f1cd8d41f8838f304177e649c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078fe9f1cd8d41f8838f304177e649c5" member_type: VOTER } }
I20260129 03:00:00.759334 23553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 078fe9f1cd8d41f8838f304177e649c5. Latest consensus state: current_term: 1 leader_uuid: "078fe9f1cd8d41f8838f304177e649c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078fe9f1cd8d41f8838f304177e649c5" member_type: VOTER } }
I20260129 03:00:00.759729 23552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:00.759963 23553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:00.769505 23559 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:00.782284 23559 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:00.784205 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:00.799019 23559 catalog_manager.cc:1357] Generated new cluster ID: 42bfd30e6c4940db8fe9aa10c3050aa4
I20260129 03:00:00.799376 23559 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:00.825538 23559 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:00.827109 23559 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:00.850225 23559 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5: Generated new TSK 0
I20260129 03:00:00.851449 23559 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:00.915690 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:00.924933 23570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:00.926373 23571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:00.928354 23573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:00.929401 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:00.930338 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:00.930608 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:00.930822 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655600930804 us; error 0 us; skew 500 ppm
I20260129 03:00:00.931592 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:00.935532 23477 webserver.cc:492] Webserver started at http://127.22.237.65:36849/ using document root <none> and password file <none>
I20260129 03:00:00.936231 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:00.936483 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:00.936822 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:00.938481 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "94882edbb8d14589b011657f18ef5e3d"
format_stamp: "Formatted at 2026-01-29 03:00:00 on dist-test-slave-1zcf"
I20260129 03:00:00.945302 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260129 03:00:00.950469 23578 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:00.951298 23477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260129 03:00:00.951653 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "94882edbb8d14589b011657f18ef5e3d"
format_stamp: "Formatted at 2026-01-29 03:00:00 on dist-test-slave-1zcf"
I20260129 03:00:00.952030 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:00.970037 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:00.971865 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:00.974674 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:00.980574 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:00.980902 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:00.981233 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:00.981460 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:01.068557 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:39387
I20260129 03:00:01.068621 23640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:39387 every 8 connection(s)
I20260129 03:00:01.076304 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:01.084784 23645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:01.087751 23646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:01.094637 23648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:01.094897 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:01.095896 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:01.096151 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:01.096359 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655601096342 us; error 0 us; skew 500 ppm
I20260129 03:00:01.097026 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:01.100063 23477 webserver.cc:492] Webserver started at http://127.22.237.66:38643/ using document root <none> and password file <none>
I20260129 03:00:01.100474 23641 heartbeater.cc:344] Connected to a master server at 127.22.237.126:41363
I20260129 03:00:01.100715 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:01.100891 23641 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:01.100979 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:01.101338 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:01.101903 23641 heartbeater.cc:507] Master 127.22.237.126:41363 requested a full tablet report, sending...
I20260129 03:00:01.102749 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/ts-1-root/instance:
uuid: "064d781d79c34238b14ca1b1f76d0e92"
format_stamp: "Formatted at 2026-01-29 03:00:01 on dist-test-slave-1zcf"
I20260129 03:00:01.104687 23513 ts_manager.cc:194] Registered new tserver with Master: 94882edbb8d14589b011657f18ef5e3d (127.22.237.65:39387)
I20260129 03:00:01.106779 23513 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50094
I20260129 03:00:01.109323 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260129 03:00:01.113430 23653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:01.114423 23477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260129 03:00:01.114768 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/ts-1-root
uuid: "064d781d79c34238b14ca1b1f76d0e92"
format_stamp: "Formatted at 2026-01-29 03:00:01 on dist-test-slave-1zcf"
I20260129 03:00:01.115082 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1769655599110862-23477-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:01.156800 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:01.158114 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:01.160180 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:01.163054 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:01.163266 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:01.163489 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:01.163641 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:01.210517 23641 heartbeater.cc:499] Master 127.22.237.126:41363 was elected leader, sending a full tablet report...
I20260129 03:00:01.255213 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.66:38735
I20260129 03:00:01.255797 23715 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.66:38735 every 8 connection(s)
I20260129 03:00:01.276471 23716 heartbeater.cc:344] Connected to a master server at 127.22.237.126:41363
I20260129 03:00:01.276831 23716 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:01.277590 23716 heartbeater.cc:507] Master 127.22.237.126:41363 requested a full tablet report, sending...
I20260129 03:00:01.279695 23513 ts_manager.cc:194] Registered new tserver with Master: 064d781d79c34238b14ca1b1f76d0e92 (127.22.237.66:38735)
I20260129 03:00:01.280637 23477 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.018247195s
I20260129 03:00:01.281044 23513 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50096
I20260129 03:00:01.312752 23513 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20260129 03:00:01.332595 23513 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260129 03:00:01.378896 23681 tablet_service.cc:1505] Processing CreateTablet for tablet 42557a15597d451597b5f4d991894cff (DEFAULT_TABLE table=test-table [id=cf9f61c0efb04fb5975bc15020d4ca5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:01.380713 23681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42557a15597d451597b5f4d991894cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:01.378896 23680 tablet_service.cc:1505] Processing CreateTablet for tablet 90364a213c8a47bfafbe69cf2a158a85 (DEFAULT_TABLE table=test-table [id=cf9f61c0efb04fb5975bc15020d4ca5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:01.381839 23680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90364a213c8a47bfafbe69cf2a158a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:01.383463 23716 heartbeater.cc:499] Master 127.22.237.126:41363 was elected leader, sending a full tablet report...
I20260129 03:00:01.389251 23606 tablet_service.cc:1505] Processing CreateTablet for tablet 42557a15597d451597b5f4d991894cff (DEFAULT_TABLE table=test-table [id=cf9f61c0efb04fb5975bc15020d4ca5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:01.390623 23606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42557a15597d451597b5f4d991894cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:01.394853 23605 tablet_service.cc:1505] Processing CreateTablet for tablet 90364a213c8a47bfafbe69cf2a158a85 (DEFAULT_TABLE table=test-table [id=cf9f61c0efb04fb5975bc15020d4ca5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:01.396292 23605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90364a213c8a47bfafbe69cf2a158a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:01.418071 23729 tablet_bootstrap.cc:492] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92: Bootstrap starting.
I20260129 03:00:01.423497 23730 tablet_bootstrap.cc:492] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d: Bootstrap starting.
I20260129 03:00:01.428735 23729 tablet_bootstrap.cc:654] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:01.428916 23730 tablet_bootstrap.cc:654] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:01.436765 23729 tablet_bootstrap.cc:492] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92: No bootstrap required, opened a new log
I20260129 03:00:01.437112 23730 tablet_bootstrap.cc:492] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d: No bootstrap required, opened a new log
I20260129 03:00:01.437271 23729 ts_tablet_manager.cc:1403] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.004s
I20260129 03:00:01.437655 23730 ts_tablet_manager.cc:1403] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260129 03:00:01.439783 23730 raft_consensus.cc:359] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:01.440263 23730 raft_consensus.cc:740] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94882edbb8d14589b011657f18ef5e3d, State: Initialized, Role: FOLLOWER
I20260129 03:00:01.440168 23729 raft_consensus.cc:359] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:01.440909 23729 raft_consensus.cc:740] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064d781d79c34238b14ca1b1f76d0e92, State: Initialized, Role: FOLLOWER
I20260129 03:00:01.440834 23730 consensus_queue.cc:260] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:01.441560 23729 consensus_queue.cc:260] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:01.447280 23730 ts_tablet_manager.cc:1434] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260129 03:00:01.448516 23730 tablet_bootstrap.cc:492] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d: Bootstrap starting.
I20260129 03:00:01.451550 23729 ts_tablet_manager.cc:1434] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260129 03:00:01.456007 23730 tablet_bootstrap.cc:654] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:01.456459 23729 tablet_bootstrap.cc:492] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92: Bootstrap starting.
I20260129 03:00:01.469383 23729 tablet_bootstrap.cc:654] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:01.477496 23729 tablet_bootstrap.cc:492] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92: No bootstrap required, opened a new log
I20260129 03:00:01.477499 23730 tablet_bootstrap.cc:492] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d: No bootstrap required, opened a new log
I20260129 03:00:01.478005 23729 ts_tablet_manager.cc:1403] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.007s
I20260129 03:00:01.478085 23730 ts_tablet_manager.cc:1403] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.005s
I20260129 03:00:01.479977 23729 raft_consensus.cc:359] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } } peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } }
I20260129 03:00:01.480456 23729 raft_consensus.cc:740] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064d781d79c34238b14ca1b1f76d0e92, State: Initialized, Role: FOLLOWER
I20260129 03:00:01.480696 23730 raft_consensus.cc:359] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } } peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } }
I20260129 03:00:01.480899 23729 consensus_queue.cc:260] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } } peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } }
I20260129 03:00:01.481341 23730 raft_consensus.cc:740] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94882edbb8d14589b011657f18ef5e3d, State: Initialized, Role: FOLLOWER
I20260129 03:00:01.482214 23730 consensus_queue.cc:260] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } } peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } }
I20260129 03:00:01.482940 23729 ts_tablet_manager.cc:1434] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260129 03:00:01.484309 23730 ts_tablet_manager.cc:1434] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260129 03:00:01.513717 23717 tablet.cc:2378] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260129 03:00:01.537744 23477 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20260129 03:00:01.538079 23477 raft_consensus.cc:423] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 0 FOLLOWER]: Emulating election...
I20260129 03:00:01.538300 23477 raft_consensus.cc:3060] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:01.542076 23477 raft_consensus.cc:697] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 1 LEADER]: Becoming Leader. State: Replica: 064d781d79c34238b14ca1b1f76d0e92, State: Running, Role: LEADER
I20260129 03:00:01.542719 23477 consensus_queue.cc:237] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } } peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } }
I20260129 03:00:01.546411 23477 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260129 03:00:01.553792 23512 catalog_manager.cc:5654] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 064d781d79c34238b14ca1b1f76d0e92 (127.22.237.66). New cstate: current_term: 1 leader_uuid: "064d781d79c34238b14ca1b1f76d0e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:01.572825 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260129 03:00:01.677024 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260129 03:00:01.880786 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260129 03:00:02.136334 23616 raft_consensus.cc:3060] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:02.144892 23740 consensus_queue.cc:1048] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:00:02.185328 23477 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260129 03:00:02.185645 23477 raft_consensus.cc:423] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 0 FOLLOWER]: Emulating election...
I20260129 03:00:02.185950 23477 raft_consensus.cc:3060] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:02.189651 23477 raft_consensus.cc:697] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 1 LEADER]: Becoming Leader. State: Replica: 064d781d79c34238b14ca1b1f76d0e92, State: Running, Role: LEADER
I20260129 03:00:02.190337 23477 consensus_queue.cc:237] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:02.192659 23477 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260129 03:00:02.197184 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260129 03:00:02.196642 23512 catalog_manager.cc:5654] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 064d781d79c34238b14ca1b1f76d0e92 (127.22.237.66). New cstate: current_term: 1 leader_uuid: "064d781d79c34238b14ca1b1f76d0e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:02.301211 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260129 03:00:02.504954 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260129 03:00:02.663348 23616 raft_consensus.cc:3060] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:02.669391 23745 consensus_queue.cc:1048] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:00:03.009253 23477 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260129 03:00:03.009500 23477 raft_consensus.cc:423] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 1 LEADER]: Emulating election...
I20260129 03:00:03.009783 23477 raft_consensus.cc:3055] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 03:00:03.009970 23477 raft_consensus.cc:740] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 064d781d79c34238b14ca1b1f76d0e92, State: Running, Role: LEADER
I20260129 03:00:03.010457 23477 consensus_queue.cc:260] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:03.011190 23477 raft_consensus.cc:3060] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:03.015087 23477 raft_consensus.cc:697] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 2 LEADER]: Becoming Leader. State: Replica: 064d781d79c34238b14ca1b1f76d0e92, State: Running, Role: LEADER
I20260129 03:00:03.015614 23477 consensus_queue.cc:237] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:03.016949 23477 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260129 03:00:03.020661 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260129 03:00:03.020306 23512 catalog_manager.cc:5654] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "064d781d79c34238b14ca1b1f76d0e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:03.124076 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260129 03:00:03.327836 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260129 03:00:03.538179 23616 raft_consensus.cc:3060] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:03.542917 23616 raft_consensus.cc:1275] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 2 FOLLOWER]: Refusing update from remote peer 064d781d79c34238b14ca1b1f76d0e92: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 03:00:03.544083 23754 consensus_queue.cc:1048] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 03:00:03.631250 23477 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260129 03:00:03.631476 23477 raft_consensus.cc:423] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [term 1 FOLLOWER]: Emulating election...
I20260129 03:00:03.631664 23477 raft_consensus.cc:3060] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:03.635294 23477 raft_consensus.cc:697] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [term 2 LEADER]: Becoming Leader. State: Replica: 94882edbb8d14589b011657f18ef5e3d, State: Running, Role: LEADER
I20260129 03:00:03.635854 23477 consensus_queue.cc:237] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } } peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } }
I20260129 03:00:03.637887 23477 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260129 03:00:03.641574 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260129 03:00:03.641469 23512 catalog_manager.cc:5654] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d reported cstate change: term changed from 1 to 2, leader changed from 064d781d79c34238b14ca1b1f76d0e92 (127.22.237.66) to 94882edbb8d14589b011657f18ef5e3d (127.22.237.65). New cstate: current_term: 2 leader_uuid: "94882edbb8d14589b011657f18ef5e3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:03.745452 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260129 03:00:03.945864 23616 raft_consensus.cc:1240] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [term 2 LEADER]: Rejecting Update request from peer 064d781d79c34238b14ca1b1f76d0e92 for earlier term 1. Current term is 2. Ops: []
I20260129 03:00:03.947134 23761 consensus_queue.cc:1059] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 03:00:03.948946 23761 consensus_queue.cc:1243] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Peer 94882edbb8d14589b011657f18ef5e3d log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260129 03:00:03.950109 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260129 03:00:03.950779 23754 raft_consensus.cc:3055] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 03:00:03.951037 23754 raft_consensus.cc:740] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 064d781d79c34238b14ca1b1f76d0e92, State: Running, Role: LEADER
I20260129 03:00:03.951541 23754 consensus_queue.cc:260] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } } peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } }
I20260129 03:00:03.952219 23754 raft_consensus.cc:3060] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:04.254211 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260129 03:00:04.258463 23691 raft_consensus.cc:1275] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 2 FOLLOWER]: Refusing update from remote peer 94882edbb8d14589b011657f18ef5e3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 03:00:04.259572 23769 consensus_queue.cc:1048] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 03:00:04.857707 23477 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20260129 03:00:04.857949 23477 raft_consensus.cc:423] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 2 FOLLOWER]: Emulating election...
I20260129 03:00:04.858229 23477 raft_consensus.cc:3060] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:04.862207 23477 raft_consensus.cc:697] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 3 LEADER]: Becoming Leader. State: Replica: 94882edbb8d14589b011657f18ef5e3d, State: Running, Role: LEADER
I20260129 03:00:04.862777 23477 consensus_queue.cc:237] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:04.864652 23477 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260129 03:00:04.868080 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260129 03:00:04.868302 23512 catalog_manager.cc:5654] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d reported cstate change: term changed from 2 to 3, leader changed from 064d781d79c34238b14ca1b1f76d0e92 (127.22.237.66) to 94882edbb8d14589b011657f18ef5e3d (127.22.237.65). New cstate: current_term: 3 leader_uuid: "94882edbb8d14589b011657f18ef5e3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:04.972275 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260129 03:00:05.032058 23616 raft_consensus.cc:1240] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 3 LEADER]: Rejecting Update request from peer 064d781d79c34238b14ca1b1f76d0e92 for earlier term 2. Current term is 3. Ops: []
I20260129 03:00:05.033239 23782 consensus_queue.cc:1059] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:00:05.034659 23782 consensus_queue.cc:1243] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Peer 94882edbb8d14589b011657f18ef5e3d log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260129 03:00:05.035046 23783 raft_consensus.cc:3055] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 2 LEADER]: Stepping down as leader of term 2
I20260129 03:00:05.035393 23783 raft_consensus.cc:740] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 064d781d79c34238b14ca1b1f76d0e92, State: Running, Role: LEADER
I20260129 03:00:05.035995 23783 consensus_queue.cc:260] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:05.036757 23783 raft_consensus.cc:3060] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:05.176087 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260129 03:00:05.354674 23691 raft_consensus.cc:1275] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 3 FOLLOWER]: Refusing update from remote peer 94882edbb8d14589b011657f18ef5e3d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 03:00:05.355800 23776 consensus_queue.cc:1048] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:00:05.479714 23477 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260129 03:00:05.480013 23477 raft_consensus.cc:423] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 3 FOLLOWER]: Emulating election...
I20260129 03:00:05.480294 23477 raft_consensus.cc:3060] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:05.484222 23477 raft_consensus.cc:697] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 4 LEADER]: Becoming Leader. State: Replica: 064d781d79c34238b14ca1b1f76d0e92, State: Running, Role: LEADER
I20260129 03:00:05.484900 23477 consensus_queue.cc:237] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:05.486228 23477 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260129 03:00:05.489686 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260129 03:00:05.489434 23512 catalog_manager.cc:5654] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 reported cstate change: term changed from 3 to 4, leader changed from 94882edbb8d14589b011657f18ef5e3d (127.22.237.65) to 064d781d79c34238b14ca1b1f76d0e92 (127.22.237.66). New cstate: current_term: 4 leader_uuid: "064d781d79c34238b14ca1b1f76d0e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:05.593238 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260129 03:00:05.796881 23477 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260129 03:00:05.879956 23616 raft_consensus.cc:3055] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 3 LEADER]: Stepping down as leader of term 3
I20260129 03:00:05.880276 23616 raft_consensus.cc:740] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 94882edbb8d14589b011657f18ef5e3d, State: Running, Role: LEADER
I20260129 03:00:05.880790 23616 consensus_queue.cc:260] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "064d781d79c34238b14ca1b1f76d0e92" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38735 } } peers { permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 } }
I20260129 03:00:05.881495 23616 raft_consensus.cc:3060] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:05.886611 23616 raft_consensus.cc:1275] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 4 FOLLOWER]: Refusing update from remote peer 064d781d79c34238b14ca1b1f76d0e92: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260129 03:00:05.887718 23783 consensus_queue.cc:1048] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94882edbb8d14589b011657f18ef5e3d" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 03:00:06.303367 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
W20260129 03:00:06.329579 23654 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260129 03:00:06.331956 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:06.332609 23477 tablet_replica.cc:333] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d: stopping tablet replica
W20260129 03:00:06.332751 23654 consensus_peers.cc:597] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 -> Peer 94882edbb8d14589b011657f18ef5e3d (127.22.237.65:39387): Couldn't send request to peer 94882edbb8d14589b011657f18ef5e3d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260129 03:00:06.333341 23477 raft_consensus.cc:2243] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [term 2 LEADER]: Raft consensus shutting down.
I20260129 03:00:06.333941 23477 raft_consensus.cc:2272] T 90364a213c8a47bfafbe69cf2a158a85 P 94882edbb8d14589b011657f18ef5e3d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:06.335762 23477 tablet_replica.cc:333] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d: stopping tablet replica
I20260129 03:00:06.336141 23477 raft_consensus.cc:2243] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:06.336460 23477 raft_consensus.cc:2272] T 42557a15597d451597b5f4d991894cff P 94882edbb8d14589b011657f18ef5e3d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:06.415212 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:00:06.424664 23477 tablet_server.cc:178] TabletServer@127.22.237.66:0 shutting down...
I20260129 03:00:06.438184 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:06.438759 23477 tablet_replica.cc:333] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92: stopping tablet replica
I20260129 03:00:06.439250 23477 raft_consensus.cc:2243] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:06.439612 23477 raft_consensus.cc:2272] T 90364a213c8a47bfafbe69cf2a158a85 P 064d781d79c34238b14ca1b1f76d0e92 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:06.440929 23477 tablet_replica.cc:333] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92: stopping tablet replica
I20260129 03:00:06.441311 23477 raft_consensus.cc:2243] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 4 LEADER]: Raft consensus shutting down.
I20260129 03:00:06.441856 23477 raft_consensus.cc:2272] T 42557a15597d451597b5f4d991894cff P 064d781d79c34238b14ca1b1f76d0e92 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:06.459836 23477 tablet_server.cc:195] TabletServer@127.22.237.66:0 shutdown complete.
I20260129 03:00:06.467435 23477 master.cc:561] Master@127.22.237.126:41363 shutting down...
I20260129 03:00:06.646611 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:06.647091 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:06.647389 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 078fe9f1cd8d41f8838f304177e649c5: stopping tablet replica
I20260129 03:00:06.664458 23477 master.cc:583] Master@127.22.237.126:41363 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (7415 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20260129 03:00:06.689455 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:45831
I20260129 03:00:06.690541 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:06.696111 23799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:06.697304 23800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:06.698179 23802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:06.698494 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:06.699983 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:06.700193 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:06.700345 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655606700329 us; error 0 us; skew 500 ppm
I20260129 03:00:06.700834 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:06.703178 23477 webserver.cc:492] Webserver started at http://127.22.237.126:35385/ using document root <none> and password file <none>
I20260129 03:00:06.703639 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:06.703811 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:06.704044 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:06.705099 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "2408b98dcc9e4f29a2b52dcd1332630d"
format_stamp: "Formatted at 2026-01-29 03:00:06 on dist-test-slave-1zcf"
I20260129 03:00:06.709789 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260129 03:00:06.713647 23807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:06.714527 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:06.714798 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "2408b98dcc9e4f29a2b52dcd1332630d"
format_stamp: "Formatted at 2026-01-29 03:00:06 on dist-test-slave-1zcf"
I20260129 03:00:06.715045 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:06.743854 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:06.744849 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:06.777572 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:45831
I20260129 03:00:06.777707 23858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:45831 every 8 connection(s)
I20260129 03:00:06.781075 23859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:06.790251 23859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d: Bootstrap starting.
I20260129 03:00:06.794122 23859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:06.798133 23859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d: No bootstrap required, opened a new log
I20260129 03:00:06.799969 23859 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2408b98dcc9e4f29a2b52dcd1332630d" member_type: VOTER }
I20260129 03:00:06.800339 23859 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:06.800582 23859 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2408b98dcc9e4f29a2b52dcd1332630d, State: Initialized, Role: FOLLOWER
I20260129 03:00:06.801075 23859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2408b98dcc9e4f29a2b52dcd1332630d" member_type: VOTER }
I20260129 03:00:06.801497 23859 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:06.801759 23859 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:06.802006 23859 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:06.805950 23859 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2408b98dcc9e4f29a2b52dcd1332630d" member_type: VOTER }
I20260129 03:00:06.806452 23859 leader_election.cc:304] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2408b98dcc9e4f29a2b52dcd1332630d; no voters:
I20260129 03:00:06.807528 23859 leader_election.cc:290] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:06.807824 23862 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:06.809049 23862 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 1 LEADER]: Becoming Leader. State: Replica: 2408b98dcc9e4f29a2b52dcd1332630d, State: Running, Role: LEADER
I20260129 03:00:06.809783 23862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2408b98dcc9e4f29a2b52dcd1332630d" member_type: VOTER }
I20260129 03:00:06.810235 23859 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:06.812361 23864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2408b98dcc9e4f29a2b52dcd1332630d. Latest consensus state: current_term: 1 leader_uuid: "2408b98dcc9e4f29a2b52dcd1332630d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2408b98dcc9e4f29a2b52dcd1332630d" member_type: VOTER } }
I20260129 03:00:06.812999 23864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:06.812377 23863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2408b98dcc9e4f29a2b52dcd1332630d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2408b98dcc9e4f29a2b52dcd1332630d" member_type: VOTER } }
I20260129 03:00:06.813814 23863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:06.818004 23867 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:06.822535 23867 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:06.823776 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:06.830442 23867 catalog_manager.cc:1357] Generated new cluster ID: 62e8f0771bee46fda9ca807da98d7d40
I20260129 03:00:06.830699 23867 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:06.868767 23867 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:06.870000 23867 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:06.881213 23867 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d: Generated new TSK 0
I20260129 03:00:06.881764 23867 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:06.889895 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:06.895394 23880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:06.896459 23881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:06.897711 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:06.897871 23883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:06.898720 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:06.898902 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:06.899055 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655606899039 us; error 0 us; skew 500 ppm
I20260129 03:00:06.899555 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:06.901615 23477 webserver.cc:492] Webserver started at http://127.22.237.65:35709/ using document root <none> and password file <none>
I20260129 03:00:06.902096 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:06.902269 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:06.902514 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:06.903545 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "990680274b8c483ebb5e7e59b9dced9c"
format_stamp: "Formatted at 2026-01-29 03:00:06 on dist-test-slave-1zcf"
I20260129 03:00:06.907748 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 03:00:06.910946 23888 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:06.911787 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:06.912110 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "990680274b8c483ebb5e7e59b9dced9c"
format_stamp: "Formatted at 2026-01-29 03:00:06 on dist-test-slave-1zcf"
I20260129 03:00:06.912400 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:06.931438 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:06.932467 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:06.933869 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:06.936022 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:06.936215 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:06.936446 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:06.936594 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:06.971477 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:40023
I20260129 03:00:06.971557 23950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:40023 every 8 connection(s)
I20260129 03:00:06.975600 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:06.982348 23955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:06.984133 23956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:06.985831 23958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:06.986248 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:06.987074 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:06.987272 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:06.987429 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655606987413 us; error 0 us; skew 500 ppm
I20260129 03:00:06.987938 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:06.990051 23951 heartbeater.cc:344] Connected to a master server at 127.22.237.126:45831
I20260129 03:00:06.990365 23477 webserver.cc:492] Webserver started at http://127.22.237.66:35613/ using document root <none> and password file <none>
I20260129 03:00:06.990458 23951 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:06.991009 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:06.991259 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:06.991335 23951 heartbeater.cc:507] Master 127.22.237.126:45831 requested a full tablet report, sending...
I20260129 03:00:06.991626 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:06.992738 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/ts-1-root/instance:
uuid: "0adaec64c2fb4fad8c61ff811bd976d9"
format_stamp: "Formatted at 2026-01-29 03:00:06 on dist-test-slave-1zcf"
I20260129 03:00:06.993364 23824 ts_manager.cc:194] Registered new tserver with Master: 990680274b8c483ebb5e7e59b9dced9c (127.22.237.65:40023)
I20260129 03:00:06.995018 23824 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35284
I20260129 03:00:06.997283 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260129 03:00:07.000700 23963 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:07.001365 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:07.001693 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/ts-1-root
uuid: "0adaec64c2fb4fad8c61ff811bd976d9"
format_stamp: "Formatted at 2026-01-29 03:00:06 on dist-test-slave-1zcf"
I20260129 03:00:07.001966 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1769655599110862-23477-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:07.017516 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:07.018678 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:07.020123 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:07.022297 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:07.022492 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:07.022711 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:07.022858 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:07.059207 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.66:46701
I20260129 03:00:07.059305 24025 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.66:46701 every 8 connection(s)
I20260129 03:00:07.071816 24026 heartbeater.cc:344] Connected to a master server at 127.22.237.126:45831
I20260129 03:00:07.072161 24026 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:07.072835 24026 heartbeater.cc:507] Master 127.22.237.126:45831 requested a full tablet report, sending...
I20260129 03:00:07.074591 23824 ts_manager.cc:194] Registered new tserver with Master: 0adaec64c2fb4fad8c61ff811bd976d9 (127.22.237.66:46701)
I20260129 03:00:07.075551 23477 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.013381669s
I20260129 03:00:07.075911 23824 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35286
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:422: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20260129 03:00:07.097400 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:00:07.097915 23951 heartbeater.cc:499] Master 127.22.237.126:45831 was elected leader, sending a full tablet report...
I20260129 03:00:07.113304 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:07.128101 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:00:07.134359 23477 tablet_server.cc:178] TabletServer@127.22.237.66:0 shutting down...
I20260129 03:00:07.147426 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:07.162094 23477 tablet_server.cc:195] TabletServer@127.22.237.66:0 shutdown complete.
I20260129 03:00:07.168192 23477 master.cc:561] Master@127.22.237.126:45831 shutting down...
I20260129 03:00:07.179948 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:07.180404 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:07.180740 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2408b98dcc9e4f29a2b52dcd1332630d: stopping tablet replica
I20260129 03:00:07.197741 23477 master.cc:583] Master@127.22.237.126:45831 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (529 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20260129 03:00:07.218354 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:39445
I20260129 03:00:07.219367 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:07.223990 24039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:07.225001 24040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:07.226384 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:07.226315 24042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:07.227300 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:07.227473 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:07.227588 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655607227577 us; error 0 us; skew 500 ppm
I20260129 03:00:07.228008 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:07.230232 23477 webserver.cc:492] Webserver started at http://127.22.237.126:42361/ using document root <none> and password file <none>
I20260129 03:00:07.230710 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:07.230863 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:07.231063 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:07.232034 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "a996e30752a54793b71e2ec74eeeb28e"
format_stamp: "Formatted at 2026-01-29 03:00:07 on dist-test-slave-1zcf"
I20260129 03:00:07.236114 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 03:00:07.239216 24047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:07.239907 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260129 03:00:07.240164 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "a996e30752a54793b71e2ec74eeeb28e"
format_stamp: "Formatted at 2026-01-29 03:00:07 on dist-test-slave-1zcf"
I20260129 03:00:07.240448 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:07.252417 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:07.253410 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:07.285830 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:39445
I20260129 03:00:07.285903 24098 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:39445 every 8 connection(s)
I20260129 03:00:07.289263 24099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:07.298208 24099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e: Bootstrap starting.
I20260129 03:00:07.302104 24099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:07.305586 24099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e: No bootstrap required, opened a new log
I20260129 03:00:07.307392 24099 raft_consensus.cc:359] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a996e30752a54793b71e2ec74eeeb28e" member_type: VOTER }
I20260129 03:00:07.307744 24099 raft_consensus.cc:385] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:07.307920 24099 raft_consensus.cc:740] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a996e30752a54793b71e2ec74eeeb28e, State: Initialized, Role: FOLLOWER
I20260129 03:00:07.308357 24099 consensus_queue.cc:260] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a996e30752a54793b71e2ec74eeeb28e" member_type: VOTER }
I20260129 03:00:07.308744 24099 raft_consensus.cc:399] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:07.308923 24099 raft_consensus.cc:493] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:07.309128 24099 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:07.313112 24099 raft_consensus.cc:515] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a996e30752a54793b71e2ec74eeeb28e" member_type: VOTER }
I20260129 03:00:07.313581 24099 leader_election.cc:304] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a996e30752a54793b71e2ec74eeeb28e; no voters:
I20260129 03:00:07.314661 24099 leader_election.cc:290] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:07.314910 24102 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:07.316229 24102 raft_consensus.cc:697] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 1 LEADER]: Becoming Leader. State: Replica: a996e30752a54793b71e2ec74eeeb28e, State: Running, Role: LEADER
I20260129 03:00:07.316843 24102 consensus_queue.cc:237] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a996e30752a54793b71e2ec74eeeb28e" member_type: VOTER }
I20260129 03:00:07.317423 24099 sys_catalog.cc:565] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:07.319293 24104 sys_catalog.cc:455] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a996e30752a54793b71e2ec74eeeb28e. Latest consensus state: current_term: 1 leader_uuid: "a996e30752a54793b71e2ec74eeeb28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a996e30752a54793b71e2ec74eeeb28e" member_type: VOTER } }
I20260129 03:00:07.319370 24103 sys_catalog.cc:455] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a996e30752a54793b71e2ec74eeeb28e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a996e30752a54793b71e2ec74eeeb28e" member_type: VOTER } }
I20260129 03:00:07.319959 24104 sys_catalog.cc:458] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:07.320011 24103 sys_catalog.cc:458] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:07.325403 24108 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:07.329824 24108 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:07.330873 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:07.337770 24108 catalog_manager.cc:1357] Generated new cluster ID: 3f33c8a34ad14a66b7d79b587c9b7424
I20260129 03:00:07.338068 24108 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:07.374729 24108 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:07.375957 24108 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:07.390377 24108 catalog_manager.cc:6027] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e: Generated new TSK 0
I20260129 03:00:07.390952 24108 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:07.397194 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:07.402693 24120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:07.403633 24121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:07.404768 24123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:07.405179 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:07.405959 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:07.406131 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:07.406247 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655607406236 us; error 0 us; skew 500 ppm
I20260129 03:00:07.406683 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:07.408728 23477 webserver.cc:492] Webserver started at http://127.22.237.65:40319/ using document root <none> and password file <none>
I20260129 03:00:07.409138 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:07.409288 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:07.409484 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:07.410521 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "c3fccde20b3d4415a1c0ddd830284edd"
format_stamp: "Formatted at 2026-01-29 03:00:07 on dist-test-slave-1zcf"
I20260129 03:00:07.414721 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260129 03:00:07.417851 24128 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:07.418619 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:07.418864 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "c3fccde20b3d4415a1c0ddd830284edd"
format_stamp: "Formatted at 2026-01-29 03:00:07 on dist-test-slave-1zcf"
I20260129 03:00:07.419147 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:07.438367 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:07.439356 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:07.440642 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:07.442864 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:07.443044 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:07.443280 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:07.443418 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:07.478199 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:45485
I20260129 03:00:07.478291 24190 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:45485 every 8 connection(s)
I20260129 03:00:07.482309 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:07.489084 24195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:07.489432 24196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:07.492014 24198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:07.492456 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:07.493474 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:07.493788 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:07.493973 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655607493954 us; error 0 us; skew 500 ppm
I20260129 03:00:07.494589 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:07.494923 24191 heartbeater.cc:344] Connected to a master server at 127.22.237.126:39445
I20260129 03:00:07.495235 24191 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:07.495961 24191 heartbeater.cc:507] Master 127.22.237.126:39445 requested a full tablet report, sending...
I20260129 03:00:07.497210 23477 webserver.cc:492] Webserver started at http://127.22.237.66:40083/ using document root <none> and password file <none>
I20260129 03:00:07.497774 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:07.497969 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:07.498083 24064 ts_manager.cc:194] Registered new tserver with Master: c3fccde20b3d4415a1c0ddd830284edd (127.22.237.65:45485)
I20260129 03:00:07.498353 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:07.499482 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/ts-1-root/instance:
uuid: "1405ae1729354621bec5b78013747ca3"
format_stamp: "Formatted at 2026-01-29 03:00:07 on dist-test-slave-1zcf"
I20260129 03:00:07.499681 24064 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37136
I20260129 03:00:07.503878 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260129 03:00:07.507179 24203 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:07.507900 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260129 03:00:07.508172 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/ts-1-root
uuid: "1405ae1729354621bec5b78013747ca3"
format_stamp: "Formatted at 2026-01-29 03:00:07 on dist-test-slave-1zcf"
I20260129 03:00:07.508424 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1769655599110862-23477-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:07.527770 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:07.528755 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:07.530138 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:07.532236 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:07.532418 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:07.532637 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:07.532788 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:07.567868 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.66:38857
I20260129 03:00:07.567950 24265 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.66:38857 every 8 connection(s)
I20260129 03:00:07.580101 24266 heartbeater.cc:344] Connected to a master server at 127.22.237.126:39445
I20260129 03:00:07.580440 24266 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:07.581074 24266 heartbeater.cc:507] Master 127.22.237.126:39445 requested a full tablet report, sending...
I20260129 03:00:07.582768 24064 ts_manager.cc:194] Registered new tserver with Master: 1405ae1729354621bec5b78013747ca3 (127.22.237.66:38857)
I20260129 03:00:07.582993 23477 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.012198104s
I20260129 03:00:07.583993 24064 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37152
I20260129 03:00:07.601879 24191 heartbeater.cc:499] Master 127.22.237.126:39445 was elected leader, sending a full tablet report...
I20260129 03:00:07.604456 24063 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260129 03:00:07.605911 24063 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260129 03:00:07.635623 24231 tablet_service.cc:1505] Processing CreateTablet for tablet 2d57e79c1748438790fd17459e8c408f (DEFAULT_TABLE table=test-table [id=5e662f2138384f64b76dca36025ea469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:07.636718 24231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d57e79c1748438790fd17459e8c408f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:07.640879 24156 tablet_service.cc:1505] Processing CreateTablet for tablet 2d57e79c1748438790fd17459e8c408f (DEFAULT_TABLE table=test-table [id=5e662f2138384f64b76dca36025ea469]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:07.641853 24156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d57e79c1748438790fd17459e8c408f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:07.649253 24279 tablet_bootstrap.cc:492] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3: Bootstrap starting.
I20260129 03:00:07.654552 24280 tablet_bootstrap.cc:492] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd: Bootstrap starting.
I20260129 03:00:07.655140 24279 tablet_bootstrap.cc:654] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:07.659165 24279 tablet_bootstrap.cc:492] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3: No bootstrap required, opened a new log
I20260129 03:00:07.659539 24279 ts_tablet_manager.cc:1403] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260129 03:00:07.660076 24280 tablet_bootstrap.cc:654] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:07.661414 24279 raft_consensus.cc:359] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 } } peers { permanent_uuid: "c3fccde20b3d4415a1c0ddd830284edd" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 45485 } }
I20260129 03:00:07.661923 24279 raft_consensus.cc:740] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1405ae1729354621bec5b78013747ca3, State: Initialized, Role: FOLLOWER
I20260129 03:00:07.662583 24279 consensus_queue.cc:260] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 } } peers { permanent_uuid: "c3fccde20b3d4415a1c0ddd830284edd" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 45485 } }
I20260129 03:00:07.664127 24280 tablet_bootstrap.cc:492] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd: No bootstrap required, opened a new log
I20260129 03:00:07.664618 24266 heartbeater.cc:499] Master 127.22.237.126:39445 was elected leader, sending a full tablet report...
I20260129 03:00:07.664927 24279 ts_tablet_manager.cc:1434] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260129 03:00:07.665349 24280 ts_tablet_manager.cc:1403] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260129 03:00:07.667589 24280 raft_consensus.cc:359] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 } } peers { permanent_uuid: "c3fccde20b3d4415a1c0ddd830284edd" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 45485 } }
I20260129 03:00:07.668193 24280 raft_consensus.cc:740] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3fccde20b3d4415a1c0ddd830284edd, State: Initialized, Role: FOLLOWER
I20260129 03:00:07.668751 24280 consensus_queue.cc:260] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 } } peers { permanent_uuid: "c3fccde20b3d4415a1c0ddd830284edd" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 45485 } }
I20260129 03:00:07.671965 24280 ts_tablet_manager.cc:1434] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260129 03:00:07.696720 23477 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260129 03:00:07.696928 23477 raft_consensus.cc:423] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 0 FOLLOWER]: Emulating election...
I20260129 03:00:07.697136 23477 raft_consensus.cc:3060] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:07.700860 23477 raft_consensus.cc:697] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 1 LEADER]: Becoming Leader. State: Replica: c3fccde20b3d4415a1c0ddd830284edd, State: Running, Role: LEADER
W20260129 03:00:07.701170 24191 heartbeater.cc:646] Failed to heartbeat to 127.22.237.126:39445 (0 consecutive failures): IO error: failing all heartbeats for tests
I20260129 03:00:07.701534 23477 consensus_queue.cc:237] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 } } peers { permanent_uuid: "c3fccde20b3d4415a1c0ddd830284edd" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 45485 } }
I20260129 03:00:07.704272 23477 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260129 03:00:07.720901 23477 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260129 03:00:07.824782 23477 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260129 03:00:08.028661 23477 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260129 03:00:08.181782 24241 raft_consensus.cc:3060] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:08.187709 24284 consensus_queue.cc:1048] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:00:08.333770 23477 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260129 03:00:08.333981 23477 raft_consensus.cc:423] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 1 LEADER]: Emulating election...
I20260129 03:00:08.334160 23477 raft_consensus.cc:3055] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 1 LEADER]: Stepping down as leader of term 1
I20260129 03:00:08.334308 23477 raft_consensus.cc:740] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3fccde20b3d4415a1c0ddd830284edd, State: Running, Role: LEADER
I20260129 03:00:08.334679 23477 consensus_queue.cc:260] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 } } peers { permanent_uuid: "c3fccde20b3d4415a1c0ddd830284edd" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 45485 } }
I20260129 03:00:08.335264 23477 raft_consensus.cc:3060] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:08.339079 23477 raft_consensus.cc:697] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 2 LEADER]: Becoming Leader. State: Replica: c3fccde20b3d4415a1c0ddd830284edd, State: Running, Role: LEADER
I20260129 03:00:08.339654 23477 consensus_queue.cc:237] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 } } peers { permanent_uuid: "c3fccde20b3d4415a1c0ddd830284edd" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 45485 } }
I20260129 03:00:08.340888 23477 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260129 03:00:08.344470 23477 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260129 03:00:08.448114 23477 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260129 03:00:08.651952 23477 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260129 03:00:08.844316 24241 raft_consensus.cc:3060] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3 [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:08.848858 24241 raft_consensus.cc:1275] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3 [term 2 FOLLOWER]: Refusing update from remote peer c3fccde20b3d4415a1c0ddd830284edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260129 03:00:08.850140 24293 consensus_queue.cc:1048] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 03:00:08.956676 23477 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260129 03:00:08.956907 23477 raft_consensus.cc:423] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 2 LEADER]: Emulating election...
I20260129 03:00:08.957122 23477 raft_consensus.cc:3055] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 2 LEADER]: Stepping down as leader of term 2
I20260129 03:00:08.957305 23477 raft_consensus.cc:740] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c3fccde20b3d4415a1c0ddd830284edd, State: Running, Role: LEADER
I20260129 03:00:08.957744 23477 consensus_queue.cc:260] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 } } peers { permanent_uuid: "c3fccde20b3d4415a1c0ddd830284edd" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 45485 } }
I20260129 03:00:08.958341 23477 raft_consensus.cc:3060] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:08.962046 23477 raft_consensus.cc:697] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 3 LEADER]: Becoming Leader. State: Replica: c3fccde20b3d4415a1c0ddd830284edd, State: Running, Role: LEADER
I20260129 03:00:08.962566 23477 consensus_queue.cc:237] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 } } peers { permanent_uuid: "c3fccde20b3d4415a1c0ddd830284edd" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 45485 } }
I20260129 03:00:08.963801 23477 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260129 03:00:08.967173 23477 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260129 03:00:09.071054 23477 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260129 03:00:09.274745 23477 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260129 03:00:09.347354 24241 raft_consensus.cc:3060] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3 [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:09.353437 24241 raft_consensus.cc:1275] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3 [term 3 FOLLOWER]: Refusing update from remote peer c3fccde20b3d4415a1c0ddd830284edd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260129 03:00:09.354888 24293 consensus_queue.cc:1048] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1405ae1729354621bec5b78013747ca3" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 03:00:09.583765 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:00:09.602336 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:09.602939 23477 tablet_replica.cc:333] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd: stopping tablet replica
I20260129 03:00:09.603473 23477 raft_consensus.cc:2243] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 3 LEADER]: Raft consensus shutting down.
I20260129 03:00:09.604050 23477 raft_consensus.cc:2272] T 2d57e79c1748438790fd17459e8c408f P c3fccde20b3d4415a1c0ddd830284edd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:09.612381 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:00:09.621040 23477 tablet_server.cc:178] TabletServer@127.22.237.66:0 shutting down...
I20260129 03:00:09.637171 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:09.637792 23477 tablet_replica.cc:333] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3: stopping tablet replica
I20260129 03:00:09.638213 23477 raft_consensus.cc:2243] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:09.638592 23477 raft_consensus.cc:2272] T 2d57e79c1748438790fd17459e8c408f P 1405ae1729354621bec5b78013747ca3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:09.656347 23477 tablet_server.cc:195] TabletServer@127.22.237.66:0 shutdown complete.
I20260129 03:00:09.664309 23477 master.cc:561] Master@127.22.237.126:39445 shutting down...
I20260129 03:00:09.678287 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:09.678898 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:09.679206 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P a996e30752a54793b71e2ec74eeeb28e: stopping tablet replica
I20260129 03:00:09.695659 23477 master.cc:583] Master@127.22.237.126:39445 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2506 ms)
[----------] 3 tests from LeadershipChangeReportingTest (10451 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20260129 03:00:09.725221 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:41965
I20260129 03:00:09.726351 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:09.731446 24306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:09.732084 24307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:09.734007 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:09.734336 24309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:09.735056 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:09.735275 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:09.735411 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655609735395 us; error 0 us; skew 500 ppm
I20260129 03:00:09.735875 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:09.738101 23477 webserver.cc:492] Webserver started at http://127.22.237.126:41629/ using document root <none> and password file <none>
I20260129 03:00:09.738559 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:09.738739 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:09.738982 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:09.740059 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "60f894be4e8b4833b82aeab62162d8ce"
format_stamp: "Formatted at 2026-01-29 03:00:09 on dist-test-slave-1zcf"
I20260129 03:00:09.744462 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260129 03:00:09.747700 24314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:09.748390 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 03:00:09.748657 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "60f894be4e8b4833b82aeab62162d8ce"
format_stamp: "Formatted at 2026-01-29 03:00:09 on dist-test-slave-1zcf"
I20260129 03:00:09.748914 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:09.777438 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:09.778729 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:09.815346 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:41965
I20260129 03:00:09.815435 24365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:41965 every 8 connection(s)
I20260129 03:00:09.819164 24366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:09.829037 24366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce: Bootstrap starting.
I20260129 03:00:09.833447 24366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:09.837509 24366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce: No bootstrap required, opened a new log
I20260129 03:00:09.839530 24366 raft_consensus.cc:359] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f894be4e8b4833b82aeab62162d8ce" member_type: VOTER }
I20260129 03:00:09.839917 24366 raft_consensus.cc:385] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:09.840142 24366 raft_consensus.cc:740] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f894be4e8b4833b82aeab62162d8ce, State: Initialized, Role: FOLLOWER
I20260129 03:00:09.840662 24366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f894be4e8b4833b82aeab62162d8ce" member_type: VOTER }
I20260129 03:00:09.841063 24366 raft_consensus.cc:399] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:09.841280 24366 raft_consensus.cc:493] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:09.841545 24366 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:09.845623 24366 raft_consensus.cc:515] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f894be4e8b4833b82aeab62162d8ce" member_type: VOTER }
I20260129 03:00:09.846156 24366 leader_election.cc:304] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60f894be4e8b4833b82aeab62162d8ce; no voters:
I20260129 03:00:09.847231 24366 leader_election.cc:290] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:09.847623 24369 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:09.848846 24369 raft_consensus.cc:697] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 1 LEADER]: Becoming Leader. State: Replica: 60f894be4e8b4833b82aeab62162d8ce, State: Running, Role: LEADER
I20260129 03:00:09.849483 24369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f894be4e8b4833b82aeab62162d8ce" member_type: VOTER }
I20260129 03:00:09.850128 24366 sys_catalog.cc:565] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:09.852301 24370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60f894be4e8b4833b82aeab62162d8ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f894be4e8b4833b82aeab62162d8ce" member_type: VOTER } }
I20260129 03:00:09.852368 24371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60f894be4e8b4833b82aeab62162d8ce. Latest consensus state: current_term: 1 leader_uuid: "60f894be4e8b4833b82aeab62162d8ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f894be4e8b4833b82aeab62162d8ce" member_type: VOTER } }
I20260129 03:00:09.853001 24371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:09.854207 24370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:09.856052 24375 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:09.863215 24375 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:09.864743 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:09.871969 24375 catalog_manager.cc:1357] Generated new cluster ID: 27770a98cbd745ebb0a28b4ddbc92ec2
I20260129 03:00:09.872188 24375 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:09.898190 24375 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:09.899371 24375 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:09.917966 24375 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce: Generated new TSK 0
I20260129 03:00:09.918514 24375 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:09.931000 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:09.936504 24387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:09.937683 24388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:09.939116 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:09.939379 24390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:09.940160 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:09.940383 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:09.940538 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655609940516 us; error 0 us; skew 500 ppm
I20260129 03:00:09.941013 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:09.943107 23477 webserver.cc:492] Webserver started at http://127.22.237.65:38039/ using document root <none> and password file <none>
I20260129 03:00:09.943565 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:09.943737 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:09.943970 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:09.945065 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "528bc946d1444fc29953c67362a2c01c"
format_stamp: "Formatted at 2026-01-29 03:00:09 on dist-test-slave-1zcf"
I20260129 03:00:09.949327 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260129 03:00:09.952565 24395 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:09.953311 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:09.953558 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "528bc946d1444fc29953c67362a2c01c"
format_stamp: "Formatted at 2026-01-29 03:00:09 on dist-test-slave-1zcf"
I20260129 03:00:09.953832 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:09.969506 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:09.970695 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:09.972077 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:09.974324 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:09.974514 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:09.974694 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:09.974819 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:10.012765 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:34217
I20260129 03:00:10.012856 24457 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:34217 every 8 connection(s)
I20260129 03:00:10.017724 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:10.025216 24462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:10.025632 24463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:10.027331 24465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:10.028331 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:10.029099 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:10.029296 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:10.029453 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655610029435 us; error 0 us; skew 500 ppm
I20260129 03:00:10.030136 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:10.031191 24458 heartbeater.cc:344] Connected to a master server at 127.22.237.126:41965
I20260129 03:00:10.031669 24458 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:10.032452 24458 heartbeater.cc:507] Master 127.22.237.126:41965 requested a full tablet report, sending...
I20260129 03:00:10.032974 23477 webserver.cc:492] Webserver started at http://127.22.237.66:38499/ using document root <none> and password file <none>
I20260129 03:00:10.033623 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:10.033888 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:10.034250 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:10.034579 24331 ts_manager.cc:194] Registered new tserver with Master: 528bc946d1444fc29953c67362a2c01c (127.22.237.65:34217)
I20260129 03:00:10.035399 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-1-root/instance:
uuid: "977b8f7894424302b337196b39cefa18"
format_stamp: "Formatted at 2026-01-29 03:00:10 on dist-test-slave-1zcf"
I20260129 03:00:10.036181 24331 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34466
I20260129 03:00:10.039880 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 03:00:10.043267 24470 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:10.044098 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260129 03:00:10.044425 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-1-root
uuid: "977b8f7894424302b337196b39cefa18"
format_stamp: "Formatted at 2026-01-29 03:00:10 on dist-test-slave-1zcf"
I20260129 03:00:10.044698 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:10.061131 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:10.062311 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:10.063673 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:10.065852 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:10.066058 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:10.066368 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:10.066516 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:10.104816 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.66:46505
I20260129 03:00:10.104912 24532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.66:46505 every 8 connection(s)
I20260129 03:00:10.109086 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:10.116950 24536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:10.117014 24537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:10.119769 24533 heartbeater.cc:344] Connected to a master server at 127.22.237.126:41965
W20260129 03:00:10.120074 24539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:10.120196 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:10.120119 24533 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:10.121371 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260129 03:00:10.121371 24533 heartbeater.cc:507] Master 127.22.237.126:41965 requested a full tablet report, sending...
W20260129 03:00:10.121807 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:10.122017 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655610121999 us; error 0 us; skew 500 ppm
I20260129 03:00:10.122722 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:10.123694 24331 ts_manager.cc:194] Registered new tserver with Master: 977b8f7894424302b337196b39cefa18 (127.22.237.66:46505)
I20260129 03:00:10.125284 23477 webserver.cc:492] Webserver started at http://127.22.237.67:46605/ using document root <none> and password file <none>
I20260129 03:00:10.125476 24331 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34470
I20260129 03:00:10.125931 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:10.126169 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:10.126469 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:10.127624 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-2-root/instance:
uuid: "655109cfcd8b4f86ae21a3dbebc81e67"
format_stamp: "Formatted at 2026-01-29 03:00:10 on dist-test-slave-1zcf"
I20260129 03:00:10.131808 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260129 03:00:10.134938 24544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:10.135634 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:10.135892 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-2-root
uuid: "655109cfcd8b4f86ae21a3dbebc81e67"
format_stamp: "Formatted at 2026-01-29 03:00:10 on dist-test-slave-1zcf"
I20260129 03:00:10.136152 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1769655599110862-23477-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:10.154158 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:10.155357 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:10.156874 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:10.159165 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:10.159358 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:10.159582 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:10.159758 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:10.224802 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.67:39571
I20260129 03:00:10.225234 24606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.67:39571 every 8 connection(s)
I20260129 03:00:10.238720 24607 heartbeater.cc:344] Connected to a master server at 127.22.237.126:41965
I20260129 03:00:10.239070 24607 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:10.239748 24607 heartbeater.cc:507] Master 127.22.237.126:41965 requested a full tablet report, sending...
I20260129 03:00:10.241506 24331 ts_manager.cc:194] Registered new tserver with Master: 655109cfcd8b4f86ae21a3dbebc81e67 (127.22.237.67:39571)
I20260129 03:00:10.242178 23477 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01320534s
I20260129 03:00:10.242847 24331 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34482
I20260129 03:00:10.266134 24331 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260129 03:00:10.267616 24331 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260129 03:00:10.302066 24423 tablet_service.cc:1505] Processing CreateTablet for tablet cdfc826ce47141dab72e7f43f4435036 (DEFAULT_TABLE table=test-table [id=04b435ed9b274cf98bce064daa7c5363]), partition=
I20260129 03:00:10.303254 24423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdfc826ce47141dab72e7f43f4435036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:10.303642 24572 tablet_service.cc:1505] Processing CreateTablet for tablet cdfc826ce47141dab72e7f43f4435036 (DEFAULT_TABLE table=test-table [id=04b435ed9b274cf98bce064daa7c5363]), partition=
I20260129 03:00:10.304637 24572 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdfc826ce47141dab72e7f43f4435036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:10.310768 24498 tablet_service.cc:1505] Processing CreateTablet for tablet cdfc826ce47141dab72e7f43f4435036 (DEFAULT_TABLE table=test-table [id=04b435ed9b274cf98bce064daa7c5363]), partition=
I20260129 03:00:10.311806 24498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdfc826ce47141dab72e7f43f4435036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:10.319814 24622 tablet_bootstrap.cc:492] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67: Bootstrap starting.
I20260129 03:00:10.326153 24623 tablet_bootstrap.cc:492] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c: Bootstrap starting.
I20260129 03:00:10.327119 24622 tablet_bootstrap.cc:654] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:10.328783 24624 tablet_bootstrap.cc:492] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18: Bootstrap starting.
I20260129 03:00:10.332571 24623 tablet_bootstrap.cc:654] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:10.334159 24622 tablet_bootstrap.cc:492] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67: No bootstrap required, opened a new log
I20260129 03:00:10.334668 24622 ts_tablet_manager.cc:1403] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260129 03:00:10.335338 24624 tablet_bootstrap.cc:654] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:10.337538 24622 raft_consensus.cc:359] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 } } peers { permanent_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 39571 } } peers { permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 } }
I20260129 03:00:10.338316 24622 raft_consensus.cc:385] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:10.338682 24622 raft_consensus.cc:740] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655109cfcd8b4f86ae21a3dbebc81e67, State: Initialized, Role: FOLLOWER
I20260129 03:00:10.339236 24623 tablet_bootstrap.cc:492] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c: No bootstrap required, opened a new log
I20260129 03:00:10.339671 24623 ts_tablet_manager.cc:1403] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260129 03:00:10.339432 24622 consensus_queue.cc:260] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 } } peers { permanent_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 39571 } } peers { permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 } }
I20260129 03:00:10.340538 24624 tablet_bootstrap.cc:492] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18: No bootstrap required, opened a new log
I20260129 03:00:10.340880 24624 ts_tablet_manager.cc:1403] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260129 03:00:10.343029 24623 raft_consensus.cc:359] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 } } peers { permanent_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 39571 } } peers { permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 } }
I20260129 03:00:10.343508 24624 raft_consensus.cc:359] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 } } peers { permanent_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 39571 } } peers { permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 } }
I20260129 03:00:10.344002 24623 raft_consensus.cc:385] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:10.344236 24624 raft_consensus.cc:385] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:10.344388 24623 raft_consensus.cc:740] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528bc946d1444fc29953c67362a2c01c, State: Initialized, Role: FOLLOWER
I20260129 03:00:10.344542 24624 raft_consensus.cc:740] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977b8f7894424302b337196b39cefa18, State: Initialized, Role: FOLLOWER
I20260129 03:00:10.345187 24624 consensus_queue.cc:260] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 } } peers { permanent_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 39571 } } peers { permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 } }
I20260129 03:00:10.345381 24623 consensus_queue.cc:260] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 } } peers { permanent_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 39571 } } peers { permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 } }
I20260129 03:00:10.348994 24533 heartbeater.cc:499] Master 127.22.237.126:41965 was elected leader, sending a full tablet report...
I20260129 03:00:10.349432 24624 ts_tablet_manager.cc:1434] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260129 03:00:10.349017 24607 heartbeater.cc:499] Master 127.22.237.126:41965 was elected leader, sending a full tablet report...
I20260129 03:00:10.350028 24622 ts_tablet_manager.cc:1434] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260129 03:00:10.352097 24458 heartbeater.cc:499] Master 127.22.237.126:41965 was elected leader, sending a full tablet report...
I20260129 03:00:10.352470 24623 ts_tablet_manager.cc:1434] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260129 03:00:10.471865 24628 raft_consensus.cc:493] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:10.472309 24628 raft_consensus.cc:515] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 } } peers { permanent_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 39571 } } peers { permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 } }
I20260129 03:00:10.474546 24628 leader_election.cc:290] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528bc946d1444fc29953c67362a2c01c (127.22.237.65:34217), 977b8f7894424302b337196b39cefa18 (127.22.237.66:46505)
I20260129 03:00:10.483032 24433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdfc826ce47141dab72e7f43f4435036" candidate_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528bc946d1444fc29953c67362a2c01c" is_pre_election: true
I20260129 03:00:10.483913 24508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdfc826ce47141dab72e7f43f4435036" candidate_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "977b8f7894424302b337196b39cefa18" is_pre_election: true
I20260129 03:00:10.483928 24433 raft_consensus.cc:2468] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655109cfcd8b4f86ae21a3dbebc81e67 in term 0.
I20260129 03:00:10.484534 24508 raft_consensus.cc:2468] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655109cfcd8b4f86ae21a3dbebc81e67 in term 0.
I20260129 03:00:10.485193 24548 leader_election.cc:304] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528bc946d1444fc29953c67362a2c01c, 655109cfcd8b4f86ae21a3dbebc81e67; no voters:
I20260129 03:00:10.485973 24628 raft_consensus.cc:2804] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 03:00:10.486228 24628 raft_consensus.cc:493] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:10.486485 24628 raft_consensus.cc:3060] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:10.491056 24628 raft_consensus.cc:515] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 } } peers { permanent_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 39571 } } peers { permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 } }
I20260129 03:00:10.492543 24628 leader_election.cc:290] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [CANDIDATE]: Term 1 election: Requested vote from peers 528bc946d1444fc29953c67362a2c01c (127.22.237.65:34217), 977b8f7894424302b337196b39cefa18 (127.22.237.66:46505)
I20260129 03:00:10.493333 24433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdfc826ce47141dab72e7f43f4435036" candidate_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528bc946d1444fc29953c67362a2c01c"
I20260129 03:00:10.493557 24508 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cdfc826ce47141dab72e7f43f4435036" candidate_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "977b8f7894424302b337196b39cefa18"
I20260129 03:00:10.493784 24433 raft_consensus.cc:3060] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:10.494058 24508 raft_consensus.cc:3060] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:10.498484 24433 raft_consensus.cc:2468] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655109cfcd8b4f86ae21a3dbebc81e67 in term 1.
I20260129 03:00:10.498623 24508 raft_consensus.cc:2468] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655109cfcd8b4f86ae21a3dbebc81e67 in term 1.
I20260129 03:00:10.499436 24548 leader_election.cc:304] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528bc946d1444fc29953c67362a2c01c, 655109cfcd8b4f86ae21a3dbebc81e67; no voters:
I20260129 03:00:10.500128 24628 raft_consensus.cc:2804] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:10.501073 24628 raft_consensus.cc:697] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 1 LEADER]: Becoming Leader. State: Replica: 655109cfcd8b4f86ae21a3dbebc81e67, State: Running, Role: LEADER
I20260129 03:00:10.501896 24628 consensus_queue.cc:237] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 } } peers { permanent_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 39571 } } peers { permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 } }
I20260129 03:00:10.508358 24330 catalog_manager.cc:5654] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655109cfcd8b4f86ae21a3dbebc81e67 (127.22.237.67). New cstate: current_term: 1 leader_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "655109cfcd8b4f86ae21a3dbebc81e67" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 39571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:10.945047 24628 consensus_queue.cc:1048] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528bc946d1444fc29953c67362a2c01c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:00:10.954952 24632 consensus_queue.cc:1048] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "977b8f7894424302b337196b39cefa18" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 46505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:00:11.597997 23477 tablet_replica.cc:333] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c: stopping tablet replica
I20260129 03:00:11.598555 23477 raft_consensus.cc:2243] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:11.598963 23477 raft_consensus.cc:2272] T cdfc826ce47141dab72e7f43f4435036 P 528bc946d1444fc29953c67362a2c01c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:11.981355 24548 consensus_queue.cc:579] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [LEADER]: The tablet replica hosted on peer 528bc946d1444fc29953c67362a2c01c has failed
W20260129 03:00:11.981900 24548 consensus_peers.cc:597] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 -> Peer 528bc946d1444fc29953c67362a2c01c (127.22.237.65:34217): Couldn't send request to peer 528bc946d1444fc29953c67362a2c01c. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260129 03:00:12.124236 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:00:12.138161 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:12.154428 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:00:12.161811 23477 tablet_server.cc:178] TabletServer@127.22.237.66:0 shutting down...
I20260129 03:00:12.174377 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:12.175004 23477 tablet_replica.cc:333] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18: stopping tablet replica
I20260129 03:00:12.175518 23477 raft_consensus.cc:2243] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:12.175921 23477 raft_consensus.cc:2272] T cdfc826ce47141dab72e7f43f4435036 P 977b8f7894424302b337196b39cefa18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:12.192705 23477 tablet_server.cc:195] TabletServer@127.22.237.66:0 shutdown complete.
I20260129 03:00:12.199826 23477 tablet_server.cc:178] TabletServer@127.22.237.67:0 shutting down...
I20260129 03:00:12.213294 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:12.213910 23477 tablet_replica.cc:333] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67: stopping tablet replica
I20260129 03:00:12.214361 23477 raft_consensus.cc:2243] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:12.215165 23477 raft_consensus.cc:2272] T cdfc826ce47141dab72e7f43f4435036 P 655109cfcd8b4f86ae21a3dbebc81e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:12.231963 23477 tablet_server.cc:195] TabletServer@127.22.237.67:0 shutdown complete.
I20260129 03:00:12.239040 23477 master.cc:561] Master@127.22.237.126:41965 shutting down...
I20260129 03:00:12.250509 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:12.251107 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:12.251492 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 60f894be4e8b4833b82aeab62162d8ce: stopping tablet replica
I20260129 03:00:12.267760 23477 master.cc:583] Master@127.22.237.126:41965 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (2566 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20260129 03:00:12.291980 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:39639
I20260129 03:00:12.293131 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:12.297973 24648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:12.298619 24649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:12.301146 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:12.301018 24651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:12.302206 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:12.302392 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:12.302512 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655612302502 us; error 0 us; skew 500 ppm
I20260129 03:00:12.303004 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:12.305160 23477 webserver.cc:492] Webserver started at http://127.22.237.126:39739/ using document root <none> and password file <none>
I20260129 03:00:12.305605 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:12.305809 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:12.306026 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:12.307008 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "2c252a3ae1164dd08123df438b83bd2a"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:12.311275 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 03:00:12.314492 24656 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:12.315177 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:12.315423 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "2c252a3ae1164dd08123df438b83bd2a"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:12.315667 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:12.326766 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:12.327688 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:12.360982 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:39639
I20260129 03:00:12.361099 24707 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:39639 every 8 connection(s)
I20260129 03:00:12.364558 24708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:12.373805 24708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a: Bootstrap starting.
I20260129 03:00:12.377851 24708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:12.381625 24708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a: No bootstrap required, opened a new log
I20260129 03:00:12.383481 24708 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c252a3ae1164dd08123df438b83bd2a" member_type: VOTER }
I20260129 03:00:12.383847 24708 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:12.384068 24708 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c252a3ae1164dd08123df438b83bd2a, State: Initialized, Role: FOLLOWER
I20260129 03:00:12.384574 24708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c252a3ae1164dd08123df438b83bd2a" member_type: VOTER }
I20260129 03:00:12.385010 24708 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:12.385222 24708 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:12.385481 24708 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:12.389600 24708 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c252a3ae1164dd08123df438b83bd2a" member_type: VOTER }
I20260129 03:00:12.390111 24708 leader_election.cc:304] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c252a3ae1164dd08123df438b83bd2a; no voters:
I20260129 03:00:12.391148 24708 leader_election.cc:290] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:12.391458 24711 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:12.392688 24711 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 1 LEADER]: Becoming Leader. State: Replica: 2c252a3ae1164dd08123df438b83bd2a, State: Running, Role: LEADER
I20260129 03:00:12.393298 24711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c252a3ae1164dd08123df438b83bd2a" member_type: VOTER }
I20260129 03:00:12.393886 24708 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:12.395867 24712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c252a3ae1164dd08123df438b83bd2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c252a3ae1164dd08123df438b83bd2a" member_type: VOTER } }
I20260129 03:00:12.395937 24713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c252a3ae1164dd08123df438b83bd2a. Latest consensus state: current_term: 1 leader_uuid: "2c252a3ae1164dd08123df438b83bd2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c252a3ae1164dd08123df438b83bd2a" member_type: VOTER } }
I20260129 03:00:12.396605 24713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:12.397105 24712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:12.398527 24717 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:12.405490 24717 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:12.407431 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:12.413934 24717 catalog_manager.cc:1357] Generated new cluster ID: 85c9df9e737440eeb988afba45588040
I20260129 03:00:12.414254 24717 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:12.442543 24717 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:12.443792 24717 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:12.460374 24717 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a: Generated new TSK 0
I20260129 03:00:12.460937 24717 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:12.473855 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:12.479310 24729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:12.480110 24730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:12.482095 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:12.482260 24732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:12.483095 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:12.483289 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:12.483441 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655612483423 us; error 0 us; skew 500 ppm
I20260129 03:00:12.483949 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:12.486238 23477 webserver.cc:492] Webserver started at http://127.22.237.65:39483/ using document root <none> and password file <none>
I20260129 03:00:12.486709 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:12.486876 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:12.487107 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:12.488106 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "eda569aab693443e84a8ed7966c78b17"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:12.492650 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260129 03:00:12.495980 24737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:12.496748 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 03:00:12.497027 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "eda569aab693443e84a8ed7966c78b17"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:12.497304 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:12.519901 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:12.521047 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:12.522671 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:12.524875 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:12.525075 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:12.525306 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:12.525457 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:12.561038 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:43269
I20260129 03:00:12.561106 24799 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:43269 every 8 connection(s)
I20260129 03:00:12.577076 24800 heartbeater.cc:344] Connected to a master server at 127.22.237.126:39639
I20260129 03:00:12.577440 24800 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:12.578133 24800 heartbeater.cc:507] Master 127.22.237.126:39639 requested a full tablet report, sending...
I20260129 03:00:12.579942 24673 ts_manager.cc:194] Registered new tserver with Master: eda569aab693443e84a8ed7966c78b17 (127.22.237.65:43269)
I20260129 03:00:12.580051 23477 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.015960276s
I20260129 03:00:12.581921 24673 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55902
I20260129 03:00:12.598961 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:00:12.615339 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:12.630371 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:00:12.637884 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260129 03:00:12.640479 23477 tablet_server.cc:93] deduplicated master addresses: 127.22.237.126:39639
W20260129 03:00:12.643266 24810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:12.644732 24811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:12.645035 24813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:12.645315 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:12.646152 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:12.646347 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:12.646507 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655612646484 us; error 0 us; skew 500 ppm
I20260129 03:00:12.646986 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:12.649145 23477 webserver.cc:492] Webserver started at http://127.22.237.65:39483/ using document root <none> and password file <none>
I20260129 03:00:12.649603 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:12.649806 23477 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260129 03:00:12.653023 23477 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:12.655470 24818 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:12.656188 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260129 03:00:12.656443 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "eda569aab693443e84a8ed7966c78b17"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:12.656689 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:12.668659 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:12.669754 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:12.671125 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:12.673281 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:12.673466 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:12.673686 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:12.673839 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260129 03:00:12.710884 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:43269
I20260129 03:00:12.710958 24880 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:43269 every 8 connection(s)
I20260129 03:00:12.715950 23477 tablet_server.cc:178] TabletServer@127.22.237.65:43269 shutting down...
I20260129 03:00:12.739032 24881 heartbeater.cc:344] Connected to a master server at 127.22.237.126:39639
I20260129 03:00:12.739615 24881 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:12.740430 24881 heartbeater.cc:507] Master 127.22.237.126:39639 requested a full tablet report, sending...
I20260129 03:00:12.742343 24673 ts_manager.cc:194] Re-registered known tserver with Master: eda569aab693443e84a8ed7966c78b17 (127.22.237.65:43269)
I20260129 03:00:12.743609 24673 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55920
I20260129 03:00:12.745841 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:12.760736 23477 tablet_server.cc:195] TabletServer@127.22.237.65:43269 shutdown complete.
I20260129 03:00:12.767590 23477 master.cc:561] Master@127.22.237.126:39639 shutting down...
I20260129 03:00:12.779708 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:12.780180 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:12.780539 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2c252a3ae1164dd08123df438b83bd2a: stopping tablet replica
I20260129 03:00:12.798234 23477 master.cc:583] Master@127.22.237.126:39639 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (527 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20260129 03:00:12.819562 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:40691,127.22.237.125:43159,127.22.237.124:40261
I20260129 03:00:12.820868 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:12.825523 24888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:12.826587 24889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:12.827750 24891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:12.828474 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:12.829212 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:12.829385 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:12.829521 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655612829505 us; error 0 us; skew 500 ppm
I20260129 03:00:12.830045 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:12.832155 23477 webserver.cc:492] Webserver started at http://127.22.237.126:38793/ using document root <none> and password file <none>
I20260129 03:00:12.832595 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:12.832758 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:12.832993 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:12.834070 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:12.838459 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260129 03:00:12.841637 24896 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:12.842375 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260129 03:00:12.842643 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:12.842907 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:12.863734 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:12.864789 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:12.898905 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:40691
I20260129 03:00:12.899000 24947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:40691 every 8 connection(s)
I20260129 03:00:12.902524 24948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:12.902565 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:12.907892 24950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:12.908319 24951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:12.910096 24948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } has no permanent_uuid. Determining permanent_uuid...
I20260129 03:00:12.911666 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:12.911734 24953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:12.912797 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:12.913075 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:12.913301 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655612913280 us; error 0 us; skew 500 ppm
I20260129 03:00:12.914055 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:12.916994 23477 webserver.cc:492] Webserver started at http://127.22.237.125:42547/ using document root <none> and password file <none>
I20260129 03:00:12.917500 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:12.917701 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:12.917935 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:12.919286 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root/instance:
uuid: "323384262be54b1395f4569b64e04543"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:12.923583 24948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } has no permanent_uuid. Determining permanent_uuid...
I20260129 03:00:12.924898 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
W20260129 03:00:12.925961 24899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111)
I20260129 03:00:12.928810 24961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260129 03:00:12.929150 24948 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.237.125:43159: Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111)
I20260129 03:00:12.929723 23477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260129 03:00:12.930006 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root
uuid: "323384262be54b1395f4569b64e04543"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:12.930272 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:12.940984 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:12.942057 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:12.976603 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.125:43159
I20260129 03:00:12.976688 25012 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.125:43159 every 8 connection(s)
I20260129 03:00:12.980209 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260129 03:00:12.980335 25013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:12.980700 24948 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } attempt: 1
W20260129 03:00:12.985877 25016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:12.988601 25017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:12.990064 25013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } has no permanent_uuid. Determining permanent_uuid...
W20260129 03:00:13.000296 25020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:13.002033 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:13.003031 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:13.003295 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:13.003484 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655613003467 us; error 0 us; skew 500 ppm
I20260129 03:00:13.004508 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:13.008692 24948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } has no permanent_uuid. Determining permanent_uuid...
I20260129 03:00:13.008886 23477 webserver.cc:492] Webserver started at http://127.22.237.124:44153/ using document root <none> and password file <none>
I20260129 03:00:13.009533 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:13.009835 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:13.009634 25013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } has no permanent_uuid. Determining permanent_uuid...
I20260129 03:00:13.010125 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:13.011662 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-2-root/instance:
uuid: "ee59203227af44a69626ad639b3960b6"
format_stamp: "Formatted at 2026-01-29 03:00:13 on dist-test-slave-1zcf"
W20260129 03:00:13.014248 24948 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.237.124:40261: Network error: Client connection negotiation failed: client connection to 127.22.237.124:40261: connect: Connection refused (error 111)
I20260129 03:00:13.017325 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260129 03:00:13.019168 25013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } has no permanent_uuid. Determining permanent_uuid...
I20260129 03:00:13.021070 25025 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:13.021898 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:13.022167 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-2-root
uuid: "ee59203227af44a69626ad639b3960b6"
format_stamp: "Formatted at 2026-01-29 03:00:13 on dist-test-slave-1zcf"
I20260129 03:00:13.022431 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260129 03:00:13.024618 25013 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.237.124:40261: Network error: Client connection negotiation failed: client connection to 127.22.237.124:40261: connect: Connection refused (error 111)
I20260129 03:00:13.041728 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:13.042804 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:13.043051 25013 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } attempt: 1
W20260129 03:00:13.047626 25013 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.237.124:40261: Network error: Client connection negotiation failed: client connection to 127.22.237.124:40261: connect: Connection refused (error 111)
I20260129 03:00:13.059732 24948 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } attempt: 1
W20260129 03:00:13.064785 24948 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.22.237.124:40261: Network error: Client connection negotiation failed: client connection to 127.22.237.124:40261: connect: Connection refused (error 111)
I20260129 03:00:13.079206 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.124:40261
I20260129 03:00:13.079291 25077 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.124:40261 every 8 connection(s)
I20260129 03:00:13.082105 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:13.082829 25078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:13.088477 25078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } has no permanent_uuid. Determining permanent_uuid...
I20260129 03:00:13.100627 25078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } has no permanent_uuid. Determining permanent_uuid...
I20260129 03:00:13.102056 25013 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } attempt: 2
I20260129 03:00:13.105228 24948 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } attempt: 2
I20260129 03:00:13.111012 25078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } has no permanent_uuid. Determining permanent_uuid...
I20260129 03:00:13.128054 25013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Bootstrap starting.
I20260129 03:00:13.133373 24948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: Bootstrap starting.
I20260129 03:00:13.133826 25013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:13.137437 25078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6: Bootstrap starting.
I20260129 03:00:13.137959 24948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:13.138836 25013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: No bootstrap required, opened a new log
I20260129 03:00:13.141782 24948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: No bootstrap required, opened a new log
I20260129 03:00:13.141566 25013 raft_consensus.cc:359] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.142408 25013 raft_consensus.cc:385] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:13.142714 25013 raft_consensus.cc:740] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323384262be54b1395f4569b64e04543, State: Initialized, Role: FOLLOWER
I20260129 03:00:13.143390 25078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:13.143427 25013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.143883 24948 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.144505 24948 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:13.144790 24948 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6b9d190288498a9d76e5abc3e2eff7, State: Initialized, Role: FOLLOWER
I20260129 03:00:13.145701 25086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:13.145644 24948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.146515 25086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:13.147560 25013 sys_catalog.cc:565] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:13.147900 25088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:13.148653 25088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:13.149593 24948 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:13.151654 25078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6: No bootstrap required, opened a new log
I20260129 03:00:13.154465 25078 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.155328 25078 raft_consensus.cc:385] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:13.155694 25078 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee59203227af44a69626ad639b3960b6, State: Initialized, Role: FOLLOWER
I20260129 03:00:13.156665 25078 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.157766 25088 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.158196 25088 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.160203 25088 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:13.162647 24988 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:13.163369 24988 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in term 0.
I20260129 03:00:13.165177 25099 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:13.165444 24899 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7; no voters:
I20260129 03:00:13.167523 25088 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 03:00:13.168020 25088 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:13.168421 25088 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:13.170198 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:13.170996 25053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in term 0.
I20260129 03:00:13.172143 25078 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:13.181039 25099 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:13.182147 25099 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.183274 25099 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.188680 24923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:13.190501 25088 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.193994 24988 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:13.195705 24988 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee59203227af44a69626ad639b3960b6 in term 0.
I20260129 03:00:13.196813 25111 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260129 03:00:13.197103 25029 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323384262be54b1395f4569b64e04543, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:13.198616 24923 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee59203227af44a69626ad639b3960b6 in current term 1: Already voted for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in this term.
I20260129 03:00:13.200353 24988 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543"
I20260129 03:00:13.201000 24988 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:13.206496 25102 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 03:00:13.207908 25102 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:13.208403 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6"
I20260129 03:00:13.208689 25102 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:13.206949 24988 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in term 1.
I20260129 03:00:13.210034 24899 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7; no voters:
I20260129 03:00:13.194957 25099 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:13.208279 25088 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 1 election: Requested vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:13.215583 25088 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:13.215538 25102 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.216763 25053 raft_consensus.cc:2393] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8d6b9d190288498a9d76e5abc3e2eff7 in current term 1: Already voted for candidate ee59203227af44a69626ad639b3960b6 in this term.
I20260129 03:00:13.218735 25102 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 1 election: Requested vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:13.216216 25088 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 1 LEADER]: Becoming Leader. State: Replica: 8d6b9d190288498a9d76e5abc3e2eff7, State: Running, Role: LEADER
I20260129 03:00:13.219700 24988 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543"
I20260129 03:00:13.220405 24988 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ee59203227af44a69626ad639b3960b6 in current term 1: Already voted for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in this term.
I20260129 03:00:13.220212 25088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:13.226035 25088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6b9d190288498a9d76e5abc3e2eff7. Latest consensus state: current_term: 1 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:13.226780 25088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:13.228181 24923 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
I20260129 03:00:13.229909 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:13.231261 25102 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260129 03:00:13.232007 25121 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:13.238919 25121 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:13.240012 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20260129 03:00:13.240309 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20260129 03:00:13.241875 25125 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260129 03:00:13.242053 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:13.242225 25125 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260129 03:00:13.248270 25053 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 1 FOLLOWER]: Refusing update from remote peer 8d6b9d190288498a9d76e5abc3e2eff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 03:00:13.248529 24988 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 1 FOLLOWER]: Refusing update from remote peer 8d6b9d190288498a9d76e5abc3e2eff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 03:00:13.250136 25116 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260129 03:00:13.250460 25128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:13.251528 25088 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260129 03:00:13.256003 25129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:13.262925 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:13.264349 25132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:13.265043 25086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6b9d190288498a9d76e5abc3e2eff7. Latest consensus state: current_term: 1 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:13.265805 25086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:13.265488 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:13.266358 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:13.266597 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655613266575 us; error 0 us; skew 500 ppm
I20260129 03:00:13.267376 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:13.266371 25102 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6b9d190288498a9d76e5abc3e2eff7. Latest consensus state: current_term: 1 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:13.268421 25102 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:13.275781 25086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:13.276502 25086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:13.280076 25102 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:13.280782 25102 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:13.281965 25116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:13.282769 25116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:13.283416 23477 webserver.cc:492] Webserver started at http://127.22.237.65:37589/ using document root <none> and password file <none>
I20260129 03:00:13.283730 25121 catalog_manager.cc:1357] Generated new cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:13.284070 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:13.284128 25121 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:13.283815 25088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:13.284428 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:13.284593 25088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:13.284783 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:13.285948 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "9920f2741247430eaf2280bba934199e"
format_stamp: "Formatted at 2026-01-29 03:00:13 on dist-test-slave-1zcf"
I20260129 03:00:13.293036 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260129 03:00:13.296766 25144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:13.297784 23477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260129 03:00:13.298144 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "9920f2741247430eaf2280bba934199e"
format_stamp: "Formatted at 2026-01-29 03:00:13 on dist-test-slave-1zcf"
I20260129 03:00:13.298514 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:13.307263 25121 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:13.308943 25121 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:13.332043 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:13.333387 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:13.336452 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:13.340348 25121 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: Generated new TSK 0
I20260129 03:00:13.340955 25121 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:13.345069 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:13.345304 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:13.345563 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:13.345729 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260129 03:00:13.384907 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:41421
I20260129 03:00:13.384979 25206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:41421 every 8 connection(s)
I20260129 03:00:13.403368 25207 heartbeater.cc:344] Connected to a master server at 127.22.237.124:40261
I20260129 03:00:13.403817 25207 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:13.404884 25207 heartbeater.cc:507] Master 127.22.237.124:40261 requested a full tablet report, sending...
I20260129 03:00:13.408035 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260129 03:00:13.408809 25043 ts_manager.cc:194] Registered new tserver with Master: 9920f2741247430eaf2280bba934199e (127.22.237.65:41421)
I20260129 03:00:13.412928 25209 heartbeater.cc:344] Connected to a master server at 127.22.237.126:40691
I20260129 03:00:13.413332 25209 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:13.414211 25209 heartbeater.cc:507] Master 127.22.237.126:40691 requested a full tablet report, sending...
I20260129 03:00:13.416255 25208 heartbeater.cc:344] Connected to a master server at 127.22.237.125:43159
I20260129 03:00:13.416608 24913 ts_manager.cc:194] Registered new tserver with Master: 9920f2741247430eaf2280bba934199e (127.22.237.65:41421)
I20260129 03:00:13.416666 25208 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:13.418895 25208 heartbeater.cc:507] Master 127.22.237.125:43159 requested a full tablet report, sending...
W20260129 03:00:13.419587 25214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:13.422509 24913 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38162
I20260129 03:00:13.423662 24978 ts_manager.cc:194] Registered new tserver with Master: 9920f2741247430eaf2280bba934199e (127.22.237.65:41421)
W20260129 03:00:13.425696 25217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:13.423292 25215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:13.427627 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:13.428457 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:13.428645 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:13.428776 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655613428764 us; error 0 us; skew 500 ppm
I20260129 03:00:13.429253 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:13.431322 23477 webserver.cc:492] Webserver started at http://127.22.237.66:35063/ using document root <none> and password file <none>
I20260129 03:00:13.431782 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:13.431984 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:13.432193 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:13.433182 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-1-root/instance:
uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
format_stamp: "Formatted at 2026-01-29 03:00:13 on dist-test-slave-1zcf"
I20260129 03:00:13.437278 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260129 03:00:13.440562 25222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:13.441781 23477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260129 03:00:13.442067 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-1-root
uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
format_stamp: "Formatted at 2026-01-29 03:00:13 on dist-test-slave-1zcf"
I20260129 03:00:13.442332 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:13.453917 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:13.455125 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:13.457093 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:13.459873 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:13.460073 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:13.460291 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:13.460443 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:13.526590 25209 heartbeater.cc:499] Master 127.22.237.126:40691 was elected leader, sending a full tablet report...
I20260129 03:00:13.532554 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.66:43759
I20260129 03:00:13.532648 25284 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.66:43759 every 8 connection(s)
I20260129 03:00:13.543994 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260129 03:00:13.557379 25287 heartbeater.cc:344] Connected to a master server at 127.22.237.126:40691
I20260129 03:00:13.558156 25287 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:13.559298 25287 heartbeater.cc:507] Master 127.22.237.126:40691 requested a full tablet report, sending...
I20260129 03:00:13.561862 24913 ts_manager.cc:194] Registered new tserver with Master: 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:13.563794 24913 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38168
W20260129 03:00:13.574405 25292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:13.583328 25286 heartbeater.cc:344] Connected to a master server at 127.22.237.125:43159
I20260129 03:00:13.583844 25286 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:13.584026 25285 heartbeater.cc:344] Connected to a master server at 127.22.237.124:40261
I20260129 03:00:13.584528 25285 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:13.584614 25286 heartbeater.cc:507] Master 127.22.237.125:43159 requested a full tablet report, sending...
W20260129 03:00:13.583125 25293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:13.585376 25285 heartbeater.cc:507] Master 127.22.237.124:40261 requested a full tablet report, sending...
W20260129 03:00:13.586182 25295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:13.587301 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:13.588126 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260129 03:00:13.588150 24978 ts_manager.cc:194] Registered new tserver with Master: 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
W20260129 03:00:13.588418 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:13.588274 25043 ts_manager.cc:194] Registered new tserver with Master: 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:13.588743 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655613588724 us; error 0 us; skew 500 ppm
I20260129 03:00:13.589480 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:13.591722 23477 webserver.cc:492] Webserver started at http://127.22.237.67:37721/ using document root <none> and password file <none>
I20260129 03:00:13.592134 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:13.592293 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:13.592485 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:13.593428 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-2-root/instance:
uuid: "4c56cc69229649979295105b5d477420"
format_stamp: "Formatted at 2026-01-29 03:00:13 on dist-test-slave-1zcf"
I20260129 03:00:13.597489 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260129 03:00:13.600697 25300 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:13.601388 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:13.601655 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-2-root
uuid: "4c56cc69229649979295105b5d477420"
format_stamp: "Formatted at 2026-01-29 03:00:13 on dist-test-slave-1zcf"
I20260129 03:00:13.601943 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:13.630520 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:13.632102 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:13.633827 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:13.635900 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:13.636106 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:13.636337 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:13.636482 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:13.666898 25287 heartbeater.cc:499] Master 127.22.237.126:40691 was elected leader, sending a full tablet report...
I20260129 03:00:13.704294 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.67:38301
I20260129 03:00:13.704510 25362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.67:38301 every 8 connection(s)
I20260129 03:00:13.726209 25363 heartbeater.cc:344] Connected to a master server at 127.22.237.124:40261
I20260129 03:00:13.726656 25363 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:13.727527 25363 heartbeater.cc:507] Master 127.22.237.124:40261 requested a full tablet report, sending...
I20260129 03:00:13.730131 25043 ts_manager.cc:194] Registered new tserver with Master: 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:13.732856 25364 heartbeater.cc:344] Connected to a master server at 127.22.237.125:43159
I20260129 03:00:13.733256 25364 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:13.733881 25365 heartbeater.cc:344] Connected to a master server at 127.22.237.126:40691
I20260129 03:00:13.734161 25364 heartbeater.cc:507] Master 127.22.237.125:43159 requested a full tablet report, sending...
I20260129 03:00:13.734272 25365 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:13.735128 25365 heartbeater.cc:507] Master 127.22.237.126:40691 requested a full tablet report, sending...
I20260129 03:00:13.736994 24913 ts_manager.cc:194] Registered new tserver with Master: 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:13.737323 24978 ts_manager.cc:194] Registered new tserver with Master: 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:13.738026 23477 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.02044926s
I20260129 03:00:13.738453 24913 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38172
I20260129 03:00:13.776577 24913 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20260129 03:00:13.778651 24913 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260129 03:00:13.840574 25365 heartbeater.cc:499] Master 127.22.237.126:40691 was elected leader, sending a full tablet report...
I20260129 03:00:13.855012 25172 tablet_service.cc:1505] Processing CreateTablet for tablet 7a63450281bb4762bdac4271b4a92365 (DEFAULT_TABLE table=test-table [id=613936357d19474fb00a3ccc9328458a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:13.855731 25170 tablet_service.cc:1505] Processing CreateTablet for tablet 332aa2d8b2044acf8f787335c4b21449 (DEFAULT_TABLE table=test-table [id=613936357d19474fb00a3ccc9328458a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:13.856503 25172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a63450281bb4762bdac4271b4a92365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:13.857125 25170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332aa2d8b2044acf8f787335c4b21449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:13.857059 25171 tablet_service.cc:1505] Processing CreateTablet for tablet 94030938de604fca93313420eff207f4 (DEFAULT_TABLE table=test-table [id=613936357d19474fb00a3ccc9328458a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:13.858487 25171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94030938de604fca93313420eff207f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:13.865587 25327 tablet_service.cc:1505] Processing CreateTablet for tablet 94030938de604fca93313420eff207f4 (DEFAULT_TABLE table=test-table [id=613936357d19474fb00a3ccc9328458a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:13.865881 25328 tablet_service.cc:1505] Processing CreateTablet for tablet 7a63450281bb4762bdac4271b4a92365 (DEFAULT_TABLE table=test-table [id=613936357d19474fb00a3ccc9328458a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:13.867089 25327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94030938de604fca93313420eff207f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:13.870743 25326 tablet_service.cc:1505] Processing CreateTablet for tablet 332aa2d8b2044acf8f787335c4b21449 (DEFAULT_TABLE table=test-table [id=613936357d19474fb00a3ccc9328458a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:13.872287 25326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332aa2d8b2044acf8f787335c4b21449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:13.875491 25250 tablet_service.cc:1505] Processing CreateTablet for tablet 7a63450281bb4762bdac4271b4a92365 (DEFAULT_TABLE table=test-table [id=613936357d19474fb00a3ccc9328458a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:13.876986 25250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a63450281bb4762bdac4271b4a92365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:13.878275 25328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a63450281bb4762bdac4271b4a92365. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:13.883523 25249 tablet_service.cc:1505] Processing CreateTablet for tablet 94030938de604fca93313420eff207f4 (DEFAULT_TABLE table=test-table [id=613936357d19474fb00a3ccc9328458a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:13.885174 25249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94030938de604fca93313420eff207f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:13.886901 25248 tablet_service.cc:1505] Processing CreateTablet for tablet 332aa2d8b2044acf8f787335c4b21449 (DEFAULT_TABLE table=test-table [id=613936357d19474fb00a3ccc9328458a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260129 03:00:13.888461 25248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332aa2d8b2044acf8f787335c4b21449. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:13.905875 25385 tablet_bootstrap.cc:492] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e: Bootstrap starting.
I20260129 03:00:13.913887 25385 tablet_bootstrap.cc:654] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:13.922544 25387 tablet_bootstrap.cc:492] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420: Bootstrap starting.
I20260129 03:00:13.924659 25386 tablet_bootstrap.cc:492] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b: Bootstrap starting.
I20260129 03:00:13.929629 25386 tablet_bootstrap.cc:654] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:13.931928 25385 tablet_bootstrap.cc:492] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e: No bootstrap required, opened a new log
I20260129 03:00:13.930938 25387 tablet_bootstrap.cc:654] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:13.934160 25385 ts_tablet_manager.cc:1403] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.012s
I20260129 03:00:13.937044 25385 raft_consensus.cc:359] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:13.937888 25385 raft_consensus.cc:385] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:13.938189 25385 raft_consensus.cc:740] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9920f2741247430eaf2280bba934199e, State: Initialized, Role: FOLLOWER
I20260129 03:00:13.938858 25385 consensus_queue.cc:260] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:13.939934 25386 tablet_bootstrap.cc:492] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b: No bootstrap required, opened a new log
I20260129 03:00:13.940399 25386 ts_tablet_manager.cc:1403] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260129 03:00:13.941187 25387 tablet_bootstrap.cc:492] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420: No bootstrap required, opened a new log
I20260129 03:00:13.941805 25387 ts_tablet_manager.cc:1403] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.000s
I20260129 03:00:13.944792 25387 raft_consensus.cc:359] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:13.945586 25387 raft_consensus.cc:385] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:13.945905 25387 raft_consensus.cc:740] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c56cc69229649979295105b5d477420, State: Initialized, Role: FOLLOWER
I20260129 03:00:13.946274 25386 raft_consensus.cc:359] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:13.947305 25386 raft_consensus.cc:385] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:13.947639 25386 raft_consensus.cc:740] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Initialized, Role: FOLLOWER
I20260129 03:00:13.946727 25387 consensus_queue.cc:260] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:13.951160 25385 ts_tablet_manager.cc:1434] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e: Time spent starting tablet: real 0.017s user 0.012s sys 0.001s
I20260129 03:00:13.953526 25385 tablet_bootstrap.cc:492] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e: Bootstrap starting.
I20260129 03:00:13.949924 25386 consensus_queue.cc:260] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:13.959149 25387 ts_tablet_manager.cc:1434] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420: Time spent starting tablet: real 0.017s user 0.008s sys 0.004s
I20260129 03:00:13.963201 25387 tablet_bootstrap.cc:492] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420: Bootstrap starting.
I20260129 03:00:13.977149 25386 ts_tablet_manager.cc:1434] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b: Time spent starting tablet: real 0.037s user 0.006s sys 0.011s
I20260129 03:00:13.977891 25385 tablet_bootstrap.cc:654] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:13.978302 25386 tablet_bootstrap.cc:492] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b: Bootstrap starting.
I20260129 03:00:13.980008 25387 tablet_bootstrap.cc:654] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:13.981109 25391 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:13.983525 25391 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:13.999814 25391 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c56cc69229649979295105b5d477420 (127.22.237.67:38301), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:14.007089 25392 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:14.008217 25392 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.012231 25393 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:14.012867 25393 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.019702 25392 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:14.033541 25393 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:14.073741 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:14.077747 25386 tablet_bootstrap.cc:654] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:14.094298 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:14.094746 25387 tablet_bootstrap.cc:492] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420: No bootstrap required, opened a new log
I20260129 03:00:14.095007 25182 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 0.
I20260129 03:00:14.095229 25387 ts_tablet_manager.cc:1403] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420: Time spent bootstrapping tablet: real 0.132s user 0.007s sys 0.006s
W20260129 03:00:14.096992 25302 leader_election.cc:343] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9920f2741247430eaf2280bba934199e (127.22.237.65:41421): Illegal state: must be running to vote when last-logged opid is not known
I20260129 03:00:14.099125 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:14.101119 25260 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 0.
I20260129 03:00:14.100325 25224 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:14.104230 25393 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 03:00:14.104655 25393 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:14.105821 25393 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:14.123216 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:14.162688 25148 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
W20260129 03:00:14.163656 25147 leader_election.cc:343] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c56cc69229649979295105b5d477420 (127.22.237.67:38301): Illegal state: must be running to vote when last-logged opid is not known
I20260129 03:00:14.163978 25393 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.167174 25393 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 1 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:14.174192 25391 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 03:00:14.175107 25391 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:14.176254 25391 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:14.193845 25391 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.195910 25391 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 1 election: Requested vote from peers 4c56cc69229649979295105b5d477420 (127.22.237.67:38301), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:14.197858 25387 raft_consensus.cc:359] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.198611 25387 raft_consensus.cc:385] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:14.198927 25387 raft_consensus.cc:740] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c56cc69229649979295105b5d477420, State: Initialized, Role: FOLLOWER
I20260129 03:00:14.199669 25386 tablet_bootstrap.cc:492] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b: No bootstrap required, opened a new log
I20260129 03:00:14.200248 25386 ts_tablet_manager.cc:1403] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b: Time spent bootstrapping tablet: real 0.222s user 0.063s sys 0.039s
I20260129 03:00:14.202204 25387 consensus_queue.cc:260] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.203505 25111 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:14.203292 25386 raft_consensus.cc:359] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.203799 25111 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: loading cluster ID for follower catalog manager: success
I20260129 03:00:14.203997 25386 raft_consensus.cc:385] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:14.204392 25386 raft_consensus.cc:740] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Initialized, Role: FOLLOWER
I20260129 03:00:14.205063 25386 consensus_queue.cc:260] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.205844 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:14.206708 25182 raft_consensus.cc:2393] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b1ed11ca9aa4d0e9474478934d90f3b in current term 1: Already voted for candidate 9920f2741247430eaf2280bba934199e in this term.
I20260129 03:00:14.207597 25386 ts_tablet_manager.cc:1434] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260129 03:00:14.208636 25386 tablet_bootstrap.cc:492] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b: Bootstrap starting.
I20260129 03:00:14.214962 25386 tablet_bootstrap.cc:654] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:14.215348 25111 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: acquiring CA information for follower catalog manager: success
I20260129 03:00:14.215520 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:14.216563 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:14.216975 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:14.219512 25385 tablet_bootstrap.cc:492] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e: No bootstrap required, opened a new log
W20260129 03:00:14.219410 25225 leader_election.cc:343] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c56cc69229649979295105b5d477420 (127.22.237.67:38301): Illegal state: must be running to vote when last-logged opid is not known
W20260129 03:00:14.220423 25225 leader_election.cc:343] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c56cc69229649979295105b5d477420 (127.22.237.67:38301): Illegal state: must be running to vote when last-logged opid is not known
W20260129 03:00:14.179291 25148 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.66:43759, user_credentials={real_user=slave}} blocked reactor thread for 74149.3us
I20260129 03:00:14.220861 25225 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b; no voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e
I20260129 03:00:14.221721 25393 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260129 03:00:14.221695 25385 ts_tablet_manager.cc:1403] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e: Time spent bootstrapping tablet: real 0.268s user 0.013s sys 0.006s
W20260129 03:00:14.233860 25147 leader_election.cc:343] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c56cc69229649979295105b5d477420 (127.22.237.67:38301): Illegal state: must be running to vote when last-logged opid is not known
I20260129 03:00:14.234984 25259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:14.236482 25387 ts_tablet_manager.cc:1434] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420: Time spent starting tablet: real 0.141s user 0.009s sys 0.000s
I20260129 03:00:14.264065 25385 raft_consensus.cc:359] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.236605 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:14.266736 25258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:14.248037 25125 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:14.268124 25392 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:14.269073 25258 raft_consensus.cc:2393] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9920f2741247430eaf2280bba934199e in current term 1: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
I20260129 03:00:14.266374 25385 raft_consensus.cc:385] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:14.236501 25259 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 0.
I20260129 03:00:14.271493 25392 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.268705 25125 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6: loading cluster ID for follower catalog manager: success
I20260129 03:00:14.274263 25385 raft_consensus.cc:740] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9920f2741247430eaf2280bba934199e, State: Initialized, Role: FOLLOWER
I20260129 03:00:14.275745 25387 tablet_bootstrap.cc:492] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420: Bootstrap starting.
I20260129 03:00:14.280874 25304 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters: 9920f2741247430eaf2280bba934199e
I20260129 03:00:14.280396 25148 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:14.289898 25391 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260129 03:00:14.290026 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:14.282366 25111 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 03:00:14.291095 25392 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:14.293437 25392 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 03:00:14.293839 25392 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:14.294418 25392 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:14.296329 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:14.297169 25386 tablet_bootstrap.cc:492] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b: No bootstrap required, opened a new log
I20260129 03:00:14.297655 25386 ts_tablet_manager.cc:1403] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b: Time spent bootstrapping tablet: real 0.089s user 0.038s sys 0.026s
I20260129 03:00:14.282028 25385 consensus_queue.cc:260] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.298520 25182 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 0.
I20260129 03:00:14.302758 25302 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:14.303902 25385 ts_tablet_manager.cc:1434] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e: Time spent starting tablet: real 0.081s user 0.007s sys 0.001s
I20260129 03:00:14.304962 25385 tablet_bootstrap.cc:492] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e: Bootstrap starting.
I20260129 03:00:14.307586 25392 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.307806 25125 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6: acquiring CA information for follower catalog manager: success
I20260129 03:00:14.309290 25403 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 03:00:14.309757 25403 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:14.310101 25403 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:14.311547 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:14.311678 25392 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
W20260129 03:00:14.313028 25304 leader_election.cc:343] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759): Illegal state: must be running to vote when last-logged opid is not known
I20260129 03:00:14.314772 25386 raft_consensus.cc:359] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.320175 25386 raft_consensus.cc:385] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260129 03:00:14.319276 25302 leader_election.cc:343] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9920f2741247430eaf2280bba934199e (127.22.237.65:41421): Illegal state: must be running to vote when last-logged opid is not known
I20260129 03:00:14.321079 25386 raft_consensus.cc:740] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Initialized, Role: FOLLOWER
I20260129 03:00:14.321941 25125 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 03:00:14.322317 25386 consensus_queue.cc:260] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.325022 25386 ts_tablet_manager.cc:1434] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260129 03:00:14.328018 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:14.328598 25260 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:14.328714 25387 tablet_bootstrap.cc:654] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:14.334877 25385 tablet_bootstrap.cc:654] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:14.337736 25403 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.339695 25403 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:14.340535 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:14.341131 25182 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:14.345886 25258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:14.346513 25258 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:14.347090 25260 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 1.
I20260129 03:00:14.349874 25304 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters: 9920f2741247430eaf2280bba934199e
I20260129 03:00:14.350737 25403 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:14.351298 25403 raft_consensus.cc:697] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 1 LEADER]: Becoming Leader. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:14.352131 25403 consensus_queue.cc:237] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.361176 25258 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 1.
I20260129 03:00:14.362928 25182 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 1.
I20260129 03:00:14.364871 25302 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:14.364981 24912 catalog_manager.cc:5654] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c56cc69229649979295105b5d477420 (127.22.237.67). New cstate: current_term: 1 leader_uuid: "4c56cc69229649979295105b5d477420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:14.366241 25403 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:14.367218 25385 tablet_bootstrap.cc:492] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e: No bootstrap required, opened a new log
I20260129 03:00:14.367578 25387 tablet_bootstrap.cc:492] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420: No bootstrap required, opened a new log
I20260129 03:00:14.367722 25385 ts_tablet_manager.cc:1403] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.000s
I20260129 03:00:14.368120 25387 ts_tablet_manager.cc:1403] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420: Time spent bootstrapping tablet: real 0.097s user 0.025s sys 0.010s
I20260129 03:00:14.368050 25403 raft_consensus.cc:697] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 1 LEADER]: Becoming Leader. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:14.369207 25403 consensus_queue.cc:237] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.371522 25387 raft_consensus.cc:359] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.372220 25387 raft_consensus.cc:385] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:14.372529 25387 raft_consensus.cc:740] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c56cc69229649979295105b5d477420, State: Initialized, Role: FOLLOWER
I20260129 03:00:14.373250 25387 consensus_queue.cc:260] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.373602 25385 raft_consensus.cc:359] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.374369 25385 raft_consensus.cc:385] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:14.374712 25385 raft_consensus.cc:740] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9920f2741247430eaf2280bba934199e, State: Initialized, Role: FOLLOWER
I20260129 03:00:14.375761 25387 ts_tablet_manager.cc:1434] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260129 03:00:14.376824 25385 consensus_queue.cc:260] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.379101 25385 ts_tablet_manager.cc:1434] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e: Time spent starting tablet: real 0.011s user 0.005s sys 0.003s
I20260129 03:00:14.405695 25403 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:14.406198 25403 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.408110 25403 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:14.408885 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:14.409164 25258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:14.409650 25182 raft_consensus.cc:2393] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c56cc69229649979295105b5d477420 in current term 1: Already voted for candidate 9920f2741247430eaf2280bba934199e in this term.
I20260129 03:00:14.409937 25258 raft_consensus.cc:2393] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c56cc69229649979295105b5d477420 in current term 1: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
I20260129 03:00:14.411136 25302 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:14.412200 25403 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:14.414162 24911 catalog_manager.cc:5654] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c56cc69229649979295105b5d477420 (127.22.237.67). New cstate: current_term: 1 leader_uuid: "4c56cc69229649979295105b5d477420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:14.416764 25403 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260129 03:00:14.437489 25403 consensus_queue.cc:1048] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:00:14.444046 25393 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:14.444593 25393 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.448510 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:14.449223 25182 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 1.
I20260129 03:00:14.451220 25181 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:14.451277 25224 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:14.446586 25393 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:14.452448 25393 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 03:00:14.452823 25393 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:14.453176 25393 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:14.453037 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:14.453881 25338 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 1.
I20260129 03:00:14.459414 25393 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.460582 25403 consensus_queue.cc:1048] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:00:14.461434 25391 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:14.461943 25391 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:14.463768 25391 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:14.465379 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:14.466159 25393 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:14.468163 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:14.469741 25148 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:14.473074 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:14.473711 25338 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:14.478209 25417 consensus_queue.cc:1048] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260129 03:00:14.483299 25338 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 2.
I20260129 03:00:14.484604 25225 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters:
I20260129 03:00:14.486068 25419 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Leader election won for term 2
I20260129 03:00:14.486617 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:14.487325 25180 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:14.488739 25403 consensus_queue.cc:1048] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:00:14.486617 25419 raft_consensus.cc:697] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 LEADER]: Becoming Leader. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:14.490468 25419 consensus_queue.cc:237] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.495976 25180 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 2.
I20260129 03:00:14.512357 25391 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 03:00:14.555730 24912 catalog_manager.cc:5654] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b reported cstate change: term changed from 0 to 2, leader changed from <none> to 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66). New cstate: current_term: 2 leader_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:14.585335 25338 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260129 03:00:14.586563 25419 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:00:14.595028 25182 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260129 03:00:14.596935 25419 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:00:14.818853 25260 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365"
dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
from {username='slave'} at 127.0.0.1:34510
I20260129 03:00:14.819582 25260 raft_consensus.cc:591] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 LEADER]: Received request to step down
I20260129 03:00:14.819922 25260 raft_consensus.cc:3055] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 LEADER]: Stepping down as leader of term 2
I20260129 03:00:14.820139 25260 raft_consensus.cc:740] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:14.820791 25260 consensus_queue.cc:260] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.821905 25260 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:14.921857 25392 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:14.922315 25392 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.923771 25392 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:14.924615 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:14.924718 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:14.925225 25260 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 3.
I20260129 03:00:14.926043 25304 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters: 9920f2741247430eaf2280bba934199e
I20260129 03:00:14.926704 25392 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 03:00:14.926985 25392 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:14.927290 25392 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:14.931540 25392 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.933073 25392 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:14.933832 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:14.934046 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:14.938577 25260 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 3.
I20260129 03:00:14.939572 25304 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters: 9920f2741247430eaf2280bba934199e
I20260129 03:00:14.940199 25392 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Leader election won for term 3
I20260129 03:00:14.940586 25392 raft_consensus.cc:697] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 3 LEADER]: Becoming Leader. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:14.941335 25392 consensus_queue.cc:237] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:14.949203 24913 catalog_manager.cc:5654] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 reported cstate change: term changed from 2 to 3, leader changed from 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66) to 4c56cc69229649979295105b5d477420 (127.22.237.67). New cstate: current_term: 3 leader_uuid: "4c56cc69229649979295105b5d477420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:15.018555 25391 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:15.019055 25391 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:15.020624 25391 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c56cc69229649979295105b5d477420 (127.22.237.67:38301), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:15.021499 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:15.021704 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:15.022392 25260 raft_consensus.cc:2393] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9920f2741247430eaf2280bba934199e in current term 3: Already voted for candidate 4c56cc69229649979295105b5d477420 in this term.
I20260129 03:00:15.023238 25148 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:15.024693 25391 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:15.034313 25391 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 03:00:15.041694 25338 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365"
dest_uuid: "4c56cc69229649979295105b5d477420"
from {username='slave'} at 127.0.0.1:47472
I20260129 03:00:15.042212 25338 raft_consensus.cc:591] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 3 LEADER]: Received request to step down
I20260129 03:00:15.042470 25338 raft_consensus.cc:3055] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 3 LEADER]: Stepping down as leader of term 3
I20260129 03:00:15.042676 25338 raft_consensus.cc:740] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:15.043227 25338 consensus_queue.cc:260] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:15.043990 25338 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:15.162118 25427 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:15.162592 25427 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:15.164069 25427 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:15.164880 25181 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:15.165047 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:15.165419 25181 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 3.
I20260129 03:00:15.165805 25338 raft_consensus.cc:2410] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2b1ed11ca9aa4d0e9474478934d90f3b for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 03:00:15.166445 25224 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:15.167227 25427 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 03:00:15.167533 25427 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:15.168023 25427 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:15.172645 25427 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:15.174101 25427 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 4 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:15.174822 25181 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:15.175071 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:15.175303 25181 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:15.175763 25338 raft_consensus.cc:2410] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b1ed11ca9aa4d0e9474478934d90f3b for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260129 03:00:15.179548 25181 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 4.
I20260129 03:00:15.180464 25224 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters: 4c56cc69229649979295105b5d477420
I20260129 03:00:15.181103 25427 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Leader election won for term 4
I20260129 03:00:15.181485 25427 raft_consensus.cc:697] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 LEADER]: Becoming Leader. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:15.182299 25427 consensus_queue.cc:237] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:15.189316 24913 catalog_manager.cc:5654] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b reported cstate change: term changed from 3 to 4, leader changed from 4c56cc69229649979295105b5d477420 (127.22.237.67) to 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66). New cstate: current_term: 4 leader_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:15.253517 25338 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "94030938de604fca93313420eff207f4"
dest_uuid: "4c56cc69229649979295105b5d477420"
from {username='slave'} at 127.0.0.1:47472
I20260129 03:00:15.254179 25338 raft_consensus.cc:591] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 1 LEADER]: Received request to step down
I20260129 03:00:15.254555 25338 raft_consensus.cc:3055] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 03:00:15.254830 25338 raft_consensus.cc:740] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:15.255558 25338 consensus_queue.cc:260] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.256798 25338 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:15.282660 25181 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260129 03:00:15.283815 25427 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:00:15.292606 25338 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260129 03:00:15.293030 25338 pending_rounds.cc:85] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420: Aborting all ops after (but not including) 2
I20260129 03:00:15.293324 25338 pending_rounds.cc:107] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260129 03:00:15.293625 25338 raft_consensus.cc:2889] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:15.295696 25427 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:00:15.424170 25427 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:15.424678 25427 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.426113 25427 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:15.426939 25181 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:15.427107 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:15.427629 25338 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 2.
I20260129 03:00:15.428383 25225 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters: 9920f2741247430eaf2280bba934199e
I20260129 03:00:15.428957 25427 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 03:00:15.429239 25427 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:15.429508 25427 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:15.433961 25427 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.434206 25391 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:15.434700 25391 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.435570 25427 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:15.436313 25391 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:15.436383 25181 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:15.436794 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:15.437057 25181 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:15.437599 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:15.437508 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:15.438412 25260 raft_consensus.cc:2393] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9920f2741247430eaf2280bba934199e in current term 2: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
I20260129 03:00:15.439503 25148 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:15.443650 25181 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 2.
I20260129 03:00:15.443650 25338 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 2.
I20260129 03:00:15.444528 25391 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 03:00:15.445005 25224 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:15.445734 25427 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Leader election won for term 2
I20260129 03:00:15.446122 25427 raft_consensus.cc:697] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 LEADER]: Becoming Leader. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:15.446781 25427 consensus_queue.cc:237] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.454959 24913 catalog_manager.cc:5654] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b reported cstate change: term changed from 1 to 2, leader changed from 4c56cc69229649979295105b5d477420 (127.22.237.67) to 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66). New cstate: current_term: 2 leader_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:15.469244 25338 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449"
dest_uuid: "4c56cc69229649979295105b5d477420"
from {username='slave'} at 127.0.0.1:47472
I20260129 03:00:15.469892 25338 raft_consensus.cc:591] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 1 LEADER]: Received request to step down
I20260129 03:00:15.470243 25338 raft_consensus.cc:3055] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 1 LEADER]: Stepping down as leader of term 1
I20260129 03:00:15.470541 25338 raft_consensus.cc:740] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:15.471256 25338 consensus_queue.cc:260] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.472445 25338 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:15.558449 25338 raft_consensus.cc:1275] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260129 03:00:15.559602 25427 consensus_queue.cc:1048] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:00:15.562219 25427 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:15.563587 25427 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.565428 25427 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:15.568132 25181 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:15.570581 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:15.573199 25338 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 2.
I20260129 03:00:15.574589 25225 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters:
I20260129 03:00:15.575690 25429 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 03:00:15.576040 25429 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:15.576386 25429 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:15.577188 25391 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:15.577323 25181 raft_consensus.cc:1275] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260129 03:00:15.578329 25391 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.580243 25391 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:15.581233 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:15.581703 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:15.581689 25446 consensus_queue.cc:1048] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260129 03:00:15.582594 25338 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 2.
I20260129 03:00:15.583670 25147 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:15.584578 25391 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 03:00:15.584966 25391 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:15.585371 25391 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:15.587106 25429 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.588243 25260 raft_consensus.cc:2393] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9920f2741247430eaf2280bba934199e in current term 2: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
I20260129 03:00:15.593972 25429 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:15.595363 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:15.595541 25181 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:15.600715 25391 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.601557 25338 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 2.
I20260129 03:00:15.602298 25181 raft_consensus.cc:2393] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b1ed11ca9aa4d0e9474478934d90f3b in current term 2: Already voted for candidate 9920f2741247430eaf2280bba934199e in this term.
I20260129 03:00:15.602716 25225 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters:
I20260129 03:00:15.604344 25429 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Leader election won for term 2
I20260129 03:00:15.604847 25429 raft_consensus.cc:697] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 LEADER]: Becoming Leader. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:15.605827 25391 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 2 election: Requested vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:15.605630 25429 consensus_queue.cc:237] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:15.606139 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:15.606456 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:15.610306 25338 raft_consensus.cc:2393] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9920f2741247430eaf2280bba934199e in current term 2: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
I20260129 03:00:15.611675 25147 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:15.612860 25391 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260129 03:00:15.617542 24913 catalog_manager.cc:5654] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b reported cstate change: term changed from 1 to 2, leader changed from 4c56cc69229649979295105b5d477420 (127.22.237.67) to 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66). New cstate: current_term: 2 leader_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:15.680217 24910 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:38194:
table { table_id: "613936357d19474fb00a3ccc9328458a" table_name: "test-table" } new_table_name: "NewTableName"
I20260129 03:00:15.695195 25181 raft_consensus.cc:1275] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260129 03:00:15.697130 25429 consensus_queue.cc:1048] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:00:15.709969 23477 master.cc:561] Master@127.22.237.126:40691 shutting down...
I20260129 03:00:15.716388 25336 raft_consensus.cc:1275] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260129 03:00:15.721563 25427 consensus_queue.cc:1048] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:00:15.709944 24913 catalog_manager.cc:5746] TS 9920f2741247430eaf2280bba934199e (127.22.237.65:41421) does not have the latest schema for tablet 332aa2d8b2044acf8f787335c4b21449 (table NewTableName [id=613936357d19474fb00a3ccc9328458a]). Expected version 1 got 0
I20260129 03:00:15.764665 25457 tablet.cc:1722] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260129 03:00:15.767474 25458 tablet.cc:1722] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260129 03:00:15.783274 25453 tablet.cc:1722] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260129 03:00:15.778945 25456 tablet.cc:1722] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260129 03:00:15.798898 25458 tablet.cc:1722] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260129 03:00:15.801913 25454 tablet.cc:1722] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260129 03:00:15.823345 25461 tablet.cc:1722] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260129 03:00:15.825999 25459 tablet.cc:1722] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260129 03:00:15.861946 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:15.863081 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:15.863549 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: stopping tablet replica
I20260129 03:00:15.876718 25454 tablet.cc:1722] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260129 03:00:15.899431 23477 master.cc:583] Master@127.22.237.126:40691 shutdown complete.
I20260129 03:00:16.006645 25467 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:16.007066 25467 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.008518 25467 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:16.009428 24988 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
W20260129 03:00:16.013002 25030 leader_election.cc:336] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691): Network error: Client connection negotiation failed: client connection to 127.22.237.126:40691: connect: Connection refused (error 111)
I20260129 03:00:16.013386 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:16.014007 25467 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260129 03:00:16.028854 25470 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:16.029277 25470 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.030786 25470 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:16.031711 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:16.032279 25053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323384262be54b1395f4569b64e04543 in term 1.
I20260129 03:00:16.033459 24964 leader_election.cc:304] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323384262be54b1395f4569b64e04543, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:16.034304 25470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260129 03:00:16.034716 25470 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:16.035128 25470 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 1 FOLLOWER]: Advancing to term 2
W20260129 03:00:16.036748 24965 leader_election.cc:336] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691): Network error: Client connection negotiation failed: client connection to 127.22.237.126:40691: connect: Connection refused (error 111)
I20260129 03:00:16.040002 25470 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.041431 25470 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 2 election: Requested vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:16.042407 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6"
I20260129 03:00:16.042959 25053 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 1 FOLLOWER]: Advancing to term 2
W20260129 03:00:16.044560 24965 leader_election.cc:336] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691): Network error: Client connection negotiation failed: client connection to 127.22.237.126:40691: connect: Connection refused (error 111)
I20260129 03:00:16.047883 25053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323384262be54b1395f4569b64e04543 in term 2.
I20260129 03:00:16.048698 24964 leader_election.cc:304] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 323384262be54b1395f4569b64e04543, ee59203227af44a69626ad639b3960b6; no voters: 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:16.049444 25470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 03:00:16.050459 25470 raft_consensus.cc:697] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 2 LEADER]: Becoming Leader. State: Replica: 323384262be54b1395f4569b64e04543, State: Running, Role: LEADER
I20260129 03:00:16.051040 25470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.054329 25473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323384262be54b1395f4569b64e04543. Latest consensus state: current_term: 2 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:16.054865 25473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:16.056316 25474 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:16.062471 25474 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=613936357d19474fb00a3ccc9328458a]
I20260129 03:00:16.067857 25474 tablet_loader.cc:96] loaded metadata for tablet 332aa2d8b2044acf8f787335c4b21449 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:16.069154 25474 tablet_loader.cc:96] loaded metadata for tablet 7a63450281bb4762bdac4271b4a92365 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:16.070196 25474 tablet_loader.cc:96] loaded metadata for tablet 94030938de604fca93313420eff207f4 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:16.071084 25474 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:16.074452 25474 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:16.074666 25474 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:16.078079 25474 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:16.081018 25474 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Loaded TSK: 0
I20260129 03:00:16.081889 25474 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:16.114490 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:16.119575 25475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:16.120186 25476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:16.121443 25478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:16.122473 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:16.123267 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:16.123474 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:16.123643 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655616123628 us; error 0 us; skew 500 ppm
I20260129 03:00:16.124140 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:16.126415 23477 webserver.cc:492] Webserver started at http://127.22.237.126:38793/ using document root <none> and password file <none>
I20260129 03:00:16.126866 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:16.127027 23477 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260129 03:00:16.130514 23477 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260129 03:00:16.133256 25483 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:16.134109 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260129 03:00:16.134418 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:16.134688 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:16.141377 25053 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 2 FOLLOWER]: Refusing update from remote peer 323384262be54b1395f4569b64e04543: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20260129 03:00:16.142679 25473 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260129 03:00:16.145735 25467 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323384262be54b1395f4569b64e04543. Latest consensus state: current_term: 2 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:16.146503 25467 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:16.151453 25470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:16.152169 25470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:16.152041 25467 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
W20260129 03:00:16.152884 24965 consensus_peers.cc:597] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 -> Peer 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691): Couldn't send request to peer 8d6b9d190288498a9d76e5abc3e2eff7. Status: Network error: Client connection negotiation failed: client connection to 127.22.237.126:40691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260129 03:00:16.153121 25467 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:16.159221 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:16.160283 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:16.189961 25364 heartbeater.cc:499] Master 127.22.237.125:43159 was elected leader, sending a full tablet report...
I20260129 03:00:16.204250 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:40691
I20260129 03:00:16.204313 25535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:40691 every 8 connection(s)
I20260129 03:00:16.207230 25208 heartbeater.cc:499] Master 127.22.237.125:43159 was elected leader, sending a full tablet report...
I20260129 03:00:16.217319 25536 sys_catalog.cc:263] Verifying existing consensus state
I20260129 03:00:16.221084 24976 catalog_manager.cc:6199] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: NewTableName [id=613936357d19474fb00a3ccc9328458a] alter complete (version 1)
I20260129 03:00:16.227176 25536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: Bootstrap starting.
I20260129 03:00:16.237520 25209 heartbeater.cc:344] Connected to a master server at 127.22.237.126:40691
I20260129 03:00:16.277855 25286 heartbeater.cc:499] Master 127.22.237.125:43159 was elected leader, sending a full tablet report...
I20260129 03:00:16.277877 25287 heartbeater.cc:344] Connected to a master server at 127.22.237.126:40691
I20260129 03:00:16.290104 25536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 03:00:16.291008 25536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: Bootstrap complete.
I20260129 03:00:16.293736 25536 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.294312 25536 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6b9d190288498a9d76e5abc3e2eff7, State: Initialized, Role: FOLLOWER
I20260129 03:00:16.294880 25536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.296438 25542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:16.296944 25542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:16.298148 25536 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:16.298817 25511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:16.302165 25365 heartbeater.cc:344] Connected to a master server at 127.22.237.126:40691
I20260129 03:00:16.322830 25542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323384262be54b1395f4569b64e04543. Latest consensus state: current_term: 2 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:16.324786 25542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:16.337145 25555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:16.338490 25555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:16.338827 25557 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:16.339139 25557 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: loading cluster ID for follower catalog manager: success
I20260129 03:00:16.346024 25557 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: acquiring CA information for follower catalog manager: success
I20260129 03:00:16.350548 25557 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 03:00:16.350129 25470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:16.351425 25470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:16.356999 25501 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9920f2741247430eaf2280bba934199e" instance_seqno: 1769655613375070) as {username='slave'} at 127.0.0.1:38364; Asking this server to re-register.
I20260129 03:00:16.360258 25209 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:16.363870 25501 ts_manager.cc:194] Registered new tserver with Master: 9920f2741247430eaf2280bba934199e (127.22.237.65:41421)
I20260129 03:00:16.392338 25501 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" instance_seqno: 1769655613512055) as {username='slave'} at 127.0.0.1:38394; Asking this server to re-register.
I20260129 03:00:16.393529 25287 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:16.396447 25501 ts_manager.cc:194] Registered new tserver with Master: 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:16.412870 25501 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4c56cc69229649979295105b5d477420" instance_seqno: 1769655613686144) as {username='slave'} at 127.0.0.1:38404; Asking this server to re-register.
I20260129 03:00:16.414147 25365 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:16.416843 25501 ts_manager.cc:194] Registered new tserver with Master: 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:16.417922 23477 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.201419368s
I20260129 03:00:16.423823 23477 master.cc:561] Master@127.22.237.125:43159 shutting down...
I20260129 03:00:16.451520 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 2 LEADER]: Raft consensus shutting down.
I20260129 03:00:16.452541 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:16.452947 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: stopping tablet replica
I20260129 03:00:16.486545 23477 master.cc:583] Master@127.22.237.125:43159 shutdown complete.
I20260129 03:00:16.632809 25555 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 323384262be54b1395f4569b64e04543)
I20260129 03:00:16.633383 25555 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.635746 25555 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
W20260129 03:00:16.650534 25486 leader_election.cc:336] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111)
I20260129 03:00:16.651839 25467 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 323384262be54b1395f4569b64e04543)
I20260129 03:00:16.652392 25467 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.654381 25467 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:16.662395 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:16.662914 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:16.663151 25053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in term 2.
I20260129 03:00:16.663570 25511 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee59203227af44a69626ad639b3960b6 in term 2.
I20260129 03:00:16.664280 25486 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543
I20260129 03:00:16.665068 25555 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 03:00:16.665266 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:16.665443 25555 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 323384262be54b1395f4569b64e04543)
I20260129 03:00:16.665938 25555 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 2 FOLLOWER]: Advancing to term 3
W20260129 03:00:16.665905 25029 leader_election.cc:336] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111)
I20260129 03:00:16.666085 25467 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 03:00:16.666661 25467 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 323384262be54b1395f4569b64e04543)
I20260129 03:00:16.666955 25467 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:16.670810 25555 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.671343 25467 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.672374 25555 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 3 election: Requested vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:16.672895 25467 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 3 election: Requested vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:16.673625 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
I20260129 03:00:16.674404 25511 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee59203227af44a69626ad639b3960b6 in current term 3: Already voted for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in this term.
I20260129 03:00:16.674595 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6"
I20260129 03:00:16.675271 25053 raft_consensus.cc:2393] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8d6b9d190288498a9d76e5abc3e2eff7 in current term 3: Already voted for candidate ee59203227af44a69626ad639b3960b6 in this term.
W20260129 03:00:16.676462 25486 leader_election.cc:336] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111)
I20260129 03:00:16.676808 25486 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7; no voters: 323384262be54b1395f4569b64e04543, ee59203227af44a69626ad639b3960b6
I20260129 03:00:16.677398 25555 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260129 03:00:16.677520 25029 leader_election.cc:336] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111)
I20260129 03:00:16.678009 25029 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:16.678735 25467 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 03:00:16.699359 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:16.704761 25565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:16.705442 25566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:16.708344 25568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:16.709065 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:16.709897 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:16.710110 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:16.710263 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655616710251 us; error 0 us; skew 500 ppm
I20260129 03:00:16.710716 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:16.713033 23477 webserver.cc:492] Webserver started at http://127.22.237.125:42547/ using document root <none> and password file <none>
I20260129 03:00:16.713485 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:16.713631 23477 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260129 03:00:16.716861 23477 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:16.719743 25573 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:16.720578 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 03:00:16.720860 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root
uuid: "323384262be54b1395f4569b64e04543"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:16.721194 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:16.744273 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:16.745591 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:16.783448 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.125:43159
I20260129 03:00:16.783543 25624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.125:43159 every 8 connection(s)
I20260129 03:00:16.791501 25625 sys_catalog.cc:263] Verifying existing consensus state
I20260129 03:00:16.796042 25625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Bootstrap starting.
I20260129 03:00:16.849253 25625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=7 ignored=0} mutations{seen=12 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 03:00:16.850168 25625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Bootstrap complete.
I20260129 03:00:16.853138 25625 raft_consensus.cc:359] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.853885 25625 raft_consensus.cc:740] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323384262be54b1395f4569b64e04543, State: Initialized, Role: FOLLOWER
I20260129 03:00:16.854620 25625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.863566 25630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:16.864441 25630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:16.866968 25625 sys_catalog.cc:565] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:16.868786 25208 heartbeater.cc:344] Connected to a master server at 127.22.237.125:43159
I20260129 03:00:16.869697 25364 heartbeater.cc:344] Connected to a master server at 127.22.237.125:43159
I20260129 03:00:16.878834 25286 heartbeater.cc:344] Connected to a master server at 127.22.237.125:43159
I20260129 03:00:16.887671 25642 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:16.887916 25642 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: loading cluster ID for follower catalog manager: success
I20260129 03:00:16.892431 25642 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: acquiring CA information for follower catalog manager: success
I20260129 03:00:16.896202 25642 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 03:00:16.941880 25467 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:16.942405 25467 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.944192 25467 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:16.945482 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 4 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:16.946261 25511 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee59203227af44a69626ad639b3960b6 in term 3.
I20260129 03:00:16.947424 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:16.948186 25467 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 03:00:16.948577 25467 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:16.948894 25467 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:16.955516 25467 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.956535 25600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 4 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:16.957381 25600 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee59203227af44a69626ad639b3960b6 in term 2.
I20260129 03:00:16.960597 25467 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 4 election: Requested vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:16.961580 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 4 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
I20260129 03:00:16.961829 25600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 4 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543"
I20260129 03:00:16.962128 25511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:16.962673 25600 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 2 FOLLOWER]: Advancing to term 4
I20260129 03:00:16.969377 25511 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee59203227af44a69626ad639b3960b6 in term 4.
I20260129 03:00:16.971050 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:16.969532 25600 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee59203227af44a69626ad639b3960b6 in term 4.
I20260129 03:00:16.972148 25467 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 03:00:16.973887 25467 raft_consensus.cc:697] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 4 LEADER]: Becoming Leader. State: Replica: ee59203227af44a69626ad639b3960b6, State: Running, Role: LEADER
I20260129 03:00:16.974805 25467 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:16.977741 25590 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9920f2741247430eaf2280bba934199e" instance_seqno: 1769655613375070) as {username='slave'} at 127.0.0.1:55954; Asking this server to re-register.
I20260129 03:00:16.978730 25589 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4c56cc69229649979295105b5d477420" instance_seqno: 1769655613686144) as {username='slave'} at 127.0.0.1:55938; Asking this server to re-register.
I20260129 03:00:16.978883 25643 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee59203227af44a69626ad639b3960b6. Latest consensus state: current_term: 4 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:16.980026 25208 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:16.979856 25643 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:16.980690 25364 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:16.983057 25644 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:16.984835 25590 ts_manager.cc:194] Registered new tserver with Master: 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:16.986891 25589 ts_manager.cc:194] Registered new tserver with Master: 9920f2741247430eaf2280bba934199e (127.22.237.65:41421)
I20260129 03:00:16.987687 25590 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" instance_seqno: 1769655613512055) as {username='slave'} at 127.0.0.1:55962; Asking this server to re-register.
I20260129 03:00:16.989876 25286 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:16.990259 25644 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=613936357d19474fb00a3ccc9328458a]
I20260129 03:00:16.994196 25590 ts_manager.cc:194] Registered new tserver with Master: 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:16.994581 23477 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.208188272s
I20260129 03:00:16.995060 25644 tablet_loader.cc:96] loaded metadata for tablet 332aa2d8b2044acf8f787335c4b21449 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:16.999722 25644 tablet_loader.cc:96] loaded metadata for tablet 7a63450281bb4762bdac4271b4a92365 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:17.000844 25644 tablet_loader.cc:96] loaded metadata for tablet 94030938de604fca93313420eff207f4 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:17.002619 25644 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:17.008095 25644 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:17.008423 25644 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:17.013314 25644 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:17.016639 25644 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6: Loaded TSK: 0
I20260129 03:00:17.017432 25644 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:17.074267 25600 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 4 FOLLOWER]: Refusing update from remote peer ee59203227af44a69626ad639b3960b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20260129 03:00:17.075562 25643 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260129 03:00:17.080610 25630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee59203227af44a69626ad639b3960b6. Latest consensus state: current_term: 4 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:17.081360 25630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:17.081533 25511 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 4 FOLLOWER]: Refusing update from remote peer ee59203227af44a69626ad639b3960b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20260129 03:00:17.082963 25467 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260129 03:00:17.085551 25643 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:17.086222 25643 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:17.089982 25555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee59203227af44a69626ad639b3960b6. Latest consensus state: current_term: 4 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:17.090729 25555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:17.091566 25650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:17.092310 25650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:17.093818 25630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:17.094458 25630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:17.206578 25285 heartbeater.cc:499] Master 127.22.237.124:40261 was elected leader, sending a full tablet report...
I20260129 03:00:17.216240 25207 heartbeater.cc:499] Master 127.22.237.124:40261 was elected leader, sending a full tablet report...
I20260129 03:00:17.216096 25363 heartbeater.cc:499] Master 127.22.237.124:40261 was elected leader, sending a full tablet report...
I20260129 03:00:17.287129 25650 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee59203227af44a69626ad639b3960b6)
I20260129 03:00:17.288126 25630 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee59203227af44a69626ad639b3960b6)
I20260129 03:00:17.288668 25630 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:17.295884 25630 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:17.310338 25651 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:17.313943 25650 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:17.322868 25650 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:17.333364 25655 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:17.313038 25651 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:17.334151 25655 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:17.337126 25655 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:17.337769 25655 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:17.338316 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:17.338297 25655 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:17.338367 25651 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:17.340592 25655 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:17.342250 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:17.345453 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:17.348917 25660 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:17.349351 25660 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:17.352278 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:17.354168 25660 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:17.354671 25660 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:17.355098 25660 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:17.356971 25660 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c56cc69229649979295105b5d477420 (127.22.237.67:38301), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:17.357889 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:17.371024 25600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:17.371236 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:17.371770 25600 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in term 4.
I20260129 03:00:17.371915 25511 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323384262be54b1395f4569b64e04543 in term 4.
I20260129 03:00:17.374142 25486 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7; no voters:
I20260129 03:00:17.375334 25650 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:17.375720 25650 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee59203227af44a69626ad639b3960b6)
I20260129 03:00:17.376081 25650 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:17.384559 25577 leader_election.cc:304] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7; no voters:
I20260129 03:00:17.386173 25630 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:17.386668 25630 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee59203227af44a69626ad639b3960b6)
I20260129 03:00:17.387061 25630 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:17.387964 25650 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:17.390486 25650 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 5 election: Requested vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:17.415987 25052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:17.599376 25660 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:17.600055 25660 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:17.856540 25660 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
W20260129 03:00:18.062975 25147 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.237.67:38301 timed out after 0.206s (SENT) [suppressed 13 similar messages]
I20260129 03:00:18.233145 25052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6"
W20260129 03:00:18.239049 25147 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.67:38301, user_credentials={real_user=slave}} blocked reactor thread for 176106us
I20260129 03:00:18.365793 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:18.366566 25180 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 4.
I20260129 03:00:18.368100 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:18.368743 25180 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 2.
I20260129 03:00:18.370062 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:18.370672 25180 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 2.
W20260129 03:00:18.373790 25576 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.124:40261, user_credentials={real_user=slave}} blocked reactor thread for 144033us
I20260129 03:00:18.375373 25052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:18.384358 25643 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:18.385102 25643 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: LEADER
W20260129 03:00:18.400064 25053 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36324 (request call id 0) took 1016 ms (client timeout 279 ms). Trace:
W20260129 03:00:18.400583 25053 rpcz_store.cc:269] 0129 03:00:17.384098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.399832 (+1015734us) service_pool.cc:212] Skipping call since client already timed out
0129 03:00:18.399987 (+ 155us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 03:00:18.403843 25182 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33718 (request call id 74) took 1054 ms. Trace:
I20260129 03:00:18.404213 25182 rpcz_store.cc:276] 0129 03:00:17.349473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.402992 (+1053519us) service_pool.cc:225] Handling call
0129 03:00:18.403336 (+ 344us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:18.403524 (+ 188us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:18.403551 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:18.403589 (+ 38us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:18.403618 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:18.403677 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.403793 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:18.411116 25203 debug-util.cc:398] Leaking SignalData structure 0x7b0800065120 after lost signal to thread 23480
W20260129 03:00:18.419920 25630 consensus_meta.cc:287] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Time spent flushing consensus metadata: real 1.032s user 0.016s sys 0.003s
I20260129 03:00:18.420382 25630 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:18.422323 25630 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 5 election: Requested vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:18.423272 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6"
I20260129 03:00:18.428085 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:18.428663 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:18.428795 25336 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 2.
I20260129 03:00:18.433009 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:18.434337 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:18.436453 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:18.438087 25302 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:18.446369 25302 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:18.446520 25666 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 03:00:18.448098 25666 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:18.452900 25302 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:18.454226 25667 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 03:00:18.572726 25257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:18.574792 25148 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:18.585695 25668 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 03:00:18.604502 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
W20260129 03:00:18.605232 25260 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34470 (request call id 7) took 240 ms (client timeout 248 ms). Trace:
W20260129 03:00:18.605607 25260 rpcz_store.cc:269] 0129 03:00:18.364834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.604265 (+239431us) service_pool.cc:225] Handling call
0129 03:00:18.605185 (+ 920us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:18.606601 25148 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:18.607817 25668 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260129 03:00:18.620820 25486 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.124:40261, user_credentials={real_user=slave}} blocked reactor thread for 387656us
I20260129 03:00:18.622452 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:18.623805 25501 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38394 (request call id 12) took 1416 ms. Trace:
I20260129 03:00:18.625409 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6"
I20260129 03:00:18.642752 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:18.644444 25148 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:18.645635 25668 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:18.624226 25501 rpcz_store.cc:276] 0129 03:00:17.207749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:17.304845 (+ 97096us) service_pool.cc:225] Handling call
0129 03:00:18.622904 (+1318059us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5400101654. stack: 00007fb19a297373 00007fb19a297270 00007fb199be9574 00000000003bff74 00000000003bff09 00007fb1a0c3c0c0 00007fb1a0cdabba 00007fb1a30f8eac 00007fb1a31f310e 00007fb1a16f24a4 00007fb1a16f2432 00007fb1a16f2361 00007fb1a16f22dc 00007fb1a16f1592 00007fb19eea934c 00007fb19eea8786
0129 03:00:18.623744 (+ 840us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26926976}
I20260129 03:00:18.696816 25510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
I20260129 03:00:18.697710 25510 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 323384262be54b1395f4569b64e04543 in current term 5: Already voted for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in this term.
I20260129 03:00:18.698961 25577 leader_election.cc:304] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 323384262be54b1395f4569b64e04543; no voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6
I20260129 03:00:18.700381 25630 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:18.753578 25667 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:18.754220 25667 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:18.756420 25667 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:18.759135 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:18.792423 25337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47450 (request call id 74) took 1472 ms. Trace:
I20260129 03:00:18.792783 25337 rpcz_store.cc:276] 0129 03:00:17.319703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:17.319875 (+ 172us) service_pool.cc:225] Handling call
0129 03:00:17.320217 (+ 342us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:17.335634 (+ 15417us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:17.335667 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:17.335711 (+ 44us) raft_consensus.cc:1645] Marking committed up to 4
0129 03:00:17.335752 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:18.792186 (+1456434us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b54001923d4. stack: 00007fb19a297373 00007fb19a297270 00007fb199be9574 00000000003bff74 00000000003bff09 00007fb1a0c3b5a1 00007fb1a0cd2a81 00007fb1a0cd14be 00007fb1a2cd5f49 00007fb1a077ad34 00007fb1a077acc2 00007fb1a077abf1 00007fb1a077ab6c 00007fb1a0779e22 00007fb19eea934c 00007fb19eea8786
0129 03:00:18.792221 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:18.792362 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34998656}
I20260129 03:00:18.798209 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:18.799942 25304 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
W20260129 03:00:18.802244 25259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34470 (request call id 9) took 436 ms (client timeout 206 ms). Trace:
I20260129 03:00:18.802395 25600 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 5 FOLLOWER]: Rejecting Update request from peer ee59203227af44a69626ad639b3960b6 for earlier term 4. Current term is 5. Ops: []
W20260129 03:00:18.802800 25259 rpcz_store.cc:269] 0129 03:00:18.365627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.802074 (+436447us) service_pool.cc:212] Skipping call since client already timed out
0129 03:00:18.802188 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 03:00:18.804119 25467 consensus_queue.cc:1059] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 }, Status: INVALID_TERM, Last received: 4.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260129 03:00:18.805167 25646 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:18.805516 25646 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee59203227af44a69626ad639b3960b6, State: Running, Role: LEADER
I20260129 03:00:18.806100 25667 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 03:00:18.806306 25646 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.17, Last appended by leader: 17, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:18.807067 25510 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 5 FOLLOWER]: Rejecting Update request from peer ee59203227af44a69626ad639b3960b6 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:18.807437 25646 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:18.806169 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
I20260129 03:00:18.808357 25511 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 323384262be54b1395f4569b64e04543 in current term 5: Already voted for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in this term.
W20260129 03:00:18.808799 25511 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40650 (request call id 1) took 204 ms (client timeout 229 ms). Trace:
W20260129 03:00:18.809154 25511 rpcz_store.cc:269] 0129 03:00:18.604445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.806018 (+201573us) service_pool.cc:225] Handling call
0129 03:00:18.808743 (+ 2725us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:18.850003 25486 leader_election.cc:336] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Timed out: RequestConsensusVote RPC to 127.22.237.125:43159 timed out after 0.226s (SENT)
I20260129 03:00:18.850565 25486 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7; no voters: 323384262be54b1395f4569b64e04543, ee59203227af44a69626ad639b3960b6
I20260129 03:00:18.928498 25600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543"
I20260129 03:00:18.929368 25600 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8d6b9d190288498a9d76e5abc3e2eff7 in current term 5: Already voted for candidate 323384262be54b1395f4569b64e04543 in this term.
I20260129 03:00:18.930804 25600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543"
I20260129 03:00:18.931684 25600 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 8d6b9d190288498a9d76e5abc3e2eff7 in current term 5: Already voted for candidate 323384262be54b1395f4569b64e04543 in this term.
I20260129 03:00:18.940564 25630 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:18.941143 25630 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:18.944409 25630 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:18.945262 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 6 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:18.945940 25053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323384262be54b1395f4569b64e04543 in term 5.
I20260129 03:00:18.947252 25576 leader_election.cc:304] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323384262be54b1395f4569b64e04543, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:18.949167 25630 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:18.949643 25630 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:18.950078 25630 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:18.960139 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 6 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:18.960992 25511 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323384262be54b1395f4569b64e04543 in term 5.
I20260129 03:00:19.036026 25667 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:19.036806 25667 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:18.957188 25630 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
W20260129 03:00:19.048206 25486 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.125:43159, user_credentials={real_user=slave}} blocked reactor thread for 198252us
I20260129 03:00:19.053177 25630 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 6 election: Requested vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:19.047391 25667 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:19.054631 25667 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:19.055172 25667 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:19.055545 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 6 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6"
I20260129 03:00:19.056106 25053 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:19.057567 25667 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:19.058611 25669 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:19.059837 25667 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:19.060312 25667 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:19.062554 25667 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:19.065508 25053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323384262be54b1395f4569b64e04543 in term 6.
I20260129 03:00:19.066004 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 6 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
I20260129 03:00:19.066708 25511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:19.066900 25576 leader_election.cc:304] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323384262be54b1395f4569b64e04543, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:19.067898 25630 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:19.074062 25511 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323384262be54b1395f4569b64e04543 in term 6.
I20260129 03:00:19.078719 25668 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:19.079324 25668 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:19.083228 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:19.083982 25337 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 4.
I20260129 03:00:19.085522 25147 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:19.081521 25668 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c56cc69229649979295105b5d477420 (127.22.237.67:38301), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:19.087249 25668 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:19.087448 25257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:19.087742 25668 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:19.088197 25668 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:19.099798 25258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34470 (request call id 8) took 734 ms (client timeout 276 ms). Trace:
W20260129 03:00:19.100278 25258 rpcz_store.cc:269] 0129 03:00:18.365240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:19.099567 (+734327us) service_pool.cc:212] Skipping call since client already timed out
0129 03:00:19.099722 (+ 155us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 03:00:19.105842 25668 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:19.108314 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:19.109130 25668 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 election: Requested vote from peers 4c56cc69229649979295105b5d477420 (127.22.237.67:38301), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:19.113817 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:19.113973 25630 raft_consensus.cc:697] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 6 LEADER]: Becoming Leader. State: Replica: 323384262be54b1395f4569b64e04543, State: Running, Role: LEADER
I20260129 03:00:19.116710 25630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.17, Last appended by leader: 17, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
W20260129 03:00:19.129132 25203 debug-util.cc:398] Leaking SignalData structure 0x7b080016a9e0 after lost signal to thread 25206
W20260129 03:00:19.129518 25203 debug-util.cc:398] Leaking SignalData structure 0x7b080016aa00 after lost signal to thread 25207
W20260129 03:00:19.130609 25203 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad920 after lost signal to thread 25284
W20260129 03:00:19.130921 25203 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad940 after lost signal to thread 25285
W20260129 03:00:19.354738 25203 debug-util.cc:398] Leaking SignalData structure 0x7b0800186020 after lost signal to thread 25362
W20260129 03:00:19.355141 25203 debug-util.cc:398] Leaking SignalData structure 0x7b080005a060 after lost signal to thread 25363
W20260129 03:00:19.356343 25203 debug-util.cc:398] Leaking SignalData structure 0x7b0800093680 after lost signal to thread 25535
I20260129 03:00:19.359269 25259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
W20260129 03:00:19.360193 25259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34478 (request call id 37) took 276 ms (client timeout 279 ms). Trace:
W20260129 03:00:19.360591 25259 rpcz_store.cc:269] 0129 03:00:19.083814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:19.358941 (+275127us) service_pool.cc:225] Handling call
0129 03:00:19.360128 (+ 1187us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:19.423506 25302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.65:41421, user_credentials={real_user=slave}} blocked reactor thread for 101270us
W20260129 03:00:19.503044 25203 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3c80 after lost signal to thread 25624
I20260129 03:00:19.579085 25672 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:19.579895 25180 raft_consensus.cc:1240] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Rejecting Update request from peer 2b1ed11ca9aa4d0e9474478934d90f3b for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:19.579705 25672 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:19.581846 25672 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:19.582893 25672 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:19.583356 25672 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:19.585501 25672 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:19.592167 25259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:19.593837 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:19.596818 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:19.601353 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:19.604060 25148 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:19.608408 25672 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:19.611588 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:19.612294 25180 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 2.
I20260129 03:00:19.616984 25455 consensus_queue.cc:1059] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:19.619549 25674 raft_consensus.cc:3055] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:19.619967 25674 raft_consensus.cc:740] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:19.620833 25674 consensus_queue.cc:260] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:19.627220 25467 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:19.627863 25467 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
W20260129 03:00:19.651135 25302 leader_election.cc:336] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9920f2741247430eaf2280bba934199e (127.22.237.65:41421): Timed out: RequestConsensusVote RPC to 127.22.237.65:41421 timed out after 0.219s (SENT)
I20260129 03:00:19.651705 25302 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:19.663568 25671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323384262be54b1395f4569b64e04543. Latest consensus state: current_term: 6 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:19.664644 25671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: LEADER
W20260129 03:00:19.668816 25224 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.237.65:41421, user_credentials={real_user=slave}} blocked reactor thread for 52104.8us
I20260129 03:00:19.669636 25467 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:19.710469 25178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:19.711701 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:19.712445 25178 raft_consensus.cc:2393] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c56cc69229649979295105b5d477420 in current term 5: Already voted for candidate 9920f2741247430eaf2280bba934199e in this term.
I20260129 03:00:19.712283 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 7 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:19.713968 25511 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee59203227af44a69626ad639b3960b6 in term 6.
I20260129 03:00:19.717635 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:19.718250 25178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:19.720273 25178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:19.721244 25178 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 2.
I20260129 03:00:19.717751 25182 raft_consensus.cc:2393] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c56cc69229649979295105b5d477420 in current term 5: Already voted for candidate 9920f2741247430eaf2280bba934199e in this term.
W20260129 03:00:19.728365 25223 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:19.732952 25467 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:19.735324 25600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 7 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:19.735883 25667 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:19.736613 25302 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:19.733485 25467 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:19.737777 25467 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 6 FOLLOWER]: Advancing to term 7
W20260129 03:00:19.739401 25224 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:19.740074 25667 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 03:00:19.743455 25302 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:19.744379 25667 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 03:00:19.744352 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:19.744863 25667 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:19.745250 25667 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:19.749850 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:19.752163 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:19.754195 25147 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:19.754537 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:19.755414 25672 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 03:00:19.756896 25147 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:19.757964 25672 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260129 03:00:19.758599 25674 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:19.765935 25467 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:19.767270 25053 raft_consensus.cc:1240] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 7 FOLLOWER]: Rejecting Update request from peer 323384262be54b1395f4569b64e04543 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:19.767172 25511 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 6 FOLLOWER]: Refusing update from remote peer 323384262be54b1395f4569b64e04543: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 6 index: 18. (index mismatch)
I20260129 03:00:19.754779 25667 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:19.770344 25679 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:19.770054 25671 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260129 03:00:19.783370 25510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 7 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
I20260129 03:00:19.785776 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:19.786135 25667 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:19.787750 25467 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 7 election: Requested vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:19.787988 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:19.789512 25302 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:19.790882 25667 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 03:00:19.793802 25671 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260129 03:00:19.794709 25630 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:19.795141 25630 raft_consensus.cc:740] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 323384262be54b1395f4569b64e04543, State: Running, Role: LEADER
I20260129 03:00:19.796376 25630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 6.18, Last appended by leader: 18, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:19.798655 25630 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:19.798905 25679 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=613936357d19474fb00a3ccc9328458a]
I20260129 03:00:19.805806 25043 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45774 (request call id 53) took 1774 ms. Trace:
I20260129 03:00:19.808614 25042 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45764 (request call id 54) took 1427 ms. Trace:
I20260129 03:00:19.809008 25042 rpcz_store.cc:276] 0129 03:00:18.381374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.381519 (+ 145us) service_pool.cc:225] Handling call
0129 03:00:19.808437 (+1426918us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":224487,"reported_tablets":3}
I20260129 03:00:19.810257 25041 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45772 (request call id 53) took 1779 ms. Trace:
I20260129 03:00:19.810721 25041 rpcz_store.cc:276] 0129 03:00:18.030287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.614979 (+584692us) service_pool.cc:225] Handling call
0129 03:00:19.809710 (+1194731us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":760,"reported_tablets":3}
I20260129 03:00:19.807008 25043 rpcz_store.cc:276] 0129 03:00:18.030653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:18.030781 (+ 128us) service_pool.cc:225] Handling call
0129 03:00:19.805317 (+1774536us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11046,"reported_tablets":3}
I20260129 03:00:19.815050 25336 raft_consensus.cc:1240] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Rejecting Update request from peer 2b1ed11ca9aa4d0e9474478934d90f3b for earlier term 2. Current term is 3. Ops: []
I20260129 03:00:19.830921 25664 consensus_queue.cc:1059] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260129 03:00:19.832074 25664 raft_consensus.cc:3055] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 LEADER]: Stepping down as leader of term 2
I20260129 03:00:19.832459 25664 raft_consensus.cc:740] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:19.833467 25664 consensus_queue.cc:260] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:19.835027 25664 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:19.835844 25679 tablet_loader.cc:96] loaded metadata for tablet 332aa2d8b2044acf8f787335c4b21449 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:19.837905 25679 tablet_loader.cc:96] loaded metadata for tablet 7a63450281bb4762bdac4271b4a92365 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:19.839326 25679 tablet_loader.cc:96] loaded metadata for tablet 94030938de604fca93313420eff207f4 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:19.840842 25600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 7 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543"
I20260129 03:00:19.841977 25600 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee59203227af44a69626ad639b3960b6 for term 7 because replica has last-logged OpId of term: 6 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 17.
I20260129 03:00:19.841883 25680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323384262be54b1395f4569b64e04543. Latest consensus state: current_term: 6 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:19.842864 25680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:19.843689 25029 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:19.844940 25467 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260129 03:00:19.840762 25679 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:19.850342 25679 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:19.850685 25679 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:19.850899 25674 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:19.851347 25674 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:19.853073 25674 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:19.854236 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:19.854568 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:19.854960 25179 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 5.
I20260129 03:00:19.855337 25336 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 4.
I20260129 03:00:19.856356 25679 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:19.856253 25224 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:19.857551 25674 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:19.857879 25674 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:19.858168 25674 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:19.861526 25679 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Loaded TSK: 0
I20260129 03:00:19.862687 25679 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:19.863854 25674 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:19.866847 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:19.867475 25179 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:19.885572 25179 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 6.
I20260129 03:00:19.886363 25674 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:19.887246 25224 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:19.887624 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:19.888247 25336 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Advancing to term 6
I20260129 03:00:19.888322 25674 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:19.888931 25674 raft_consensus.cc:697] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 LEADER]: Becoming Leader. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:19.889834 25674 consensus_queue.cc:237] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:19.897308 25336 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 6.
I20260129 03:00:19.979578 25336 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260129 03:00:19.980850 25674 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:19.981155 25672 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:19.981731 25672 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:19.983882 25672 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:19.985738 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:19.986181 25630 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:19.986307 25260 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 3.
I20260129 03:00:19.986685 25630 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:19.988577 25148 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:19.988891 25630 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:19.990345 25672 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 03:00:19.990789 25672 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:19.990669 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 8 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:19.991500 25672 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:19.991552 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:19.992543 25337 raft_consensus.cc:2393] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9920f2741247430eaf2280bba934199e in current term 3: Already voted for candidate 4c56cc69229649979295105b5d477420 in this term.
I20260129 03:00:19.990751 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 8 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:19.993599 25053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323384262be54b1395f4569b64e04543 in term 7.
I20260129 03:00:19.995071 25576 leader_election.cc:304] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 323384262be54b1395f4569b64e04543, ee59203227af44a69626ad639b3960b6; no voters: 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:19.996345 25630 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:19.997030 25630 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:19.996982 25680 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 323384262be54b1395f4569b64e04543)
I20260129 03:00:19.997710 25630 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:19.997753 25680 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:20.002257 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 7 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:20.002240 25600 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 7 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:20.003181 25053 raft_consensus.cc:2393] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d6b9d190288498a9d76e5abc3e2eff7 in current term 7: Already voted for candidate ee59203227af44a69626ad639b3960b6 in this term.
I20260129 03:00:19.999992 25680 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:20.005565 25630 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:20.006985 25600 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d6b9d190288498a9d76e5abc3e2eff7 for earlier term 7. Current term is 8.
I20260129 03:00:20.008687 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 8 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
I20260129 03:00:20.009286 25511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 6 FOLLOWER]: Advancing to term 8
I20260129 03:00:20.010267 25486 leader_election.cc:400] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d6b9d190288498a9d76e5abc3e2eff7 for earlier term 7. Current term is 8.
I20260129 03:00:20.010617 25486 leader_election.cc:403] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260129 03:00:20.006214 25672 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:20.018669 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:20.020501 25511 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323384262be54b1395f4569b64e04543 in term 8.
I20260129 03:00:20.021176 25680 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d6b9d190288498a9d76e5abc3e2eff7 for earlier term 7. Current term is 8.
I20260129 03:00:20.023185 25577 leader_election.cc:304] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7; no voters:
I20260129 03:00:20.021193 25179 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260129 03:00:20.024080 25671 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:20.024760 25671 raft_consensus.cc:697] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 LEADER]: Becoming Leader. State: Replica: 323384262be54b1395f4569b64e04543, State: Running, Role: LEADER
I20260129 03:00:20.025395 25674 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260129 03:00:20.025589 25671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 6.18, Last appended by leader: 18, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:20.031507 25260 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 3.
I20260129 03:00:20.032851 25678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323384262be54b1395f4569b64e04543. Latest consensus state: current_term: 8 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:20.033200 25148 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:20.033679 25678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:20.034421 25679 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:20.040606 25672 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 election: Requested vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:20.040546 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:20.041688 25337 raft_consensus.cc:2393] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9920f2741247430eaf2280bba934199e in current term 3: Already voted for candidate 4c56cc69229649979295105b5d477420 in this term.
I20260129 03:00:20.040474 25679 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=613936357d19474fb00a3ccc9328458a]
I20260129 03:00:20.043043 25672 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Leader election won for term 3
I20260129 03:00:20.043534 25672 raft_consensus.cc:697] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 3 LEADER]: Becoming Leader. State: Replica: 9920f2741247430eaf2280bba934199e, State: Running, Role: LEADER
I20260129 03:00:20.045680 25672 consensus_queue.cc:237] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:20.048804 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 8 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6"
I20260129 03:00:20.049458 25053 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:20.064301 25053 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323384262be54b1395f4569b64e04543 in term 8.
I20260129 03:00:20.068195 25630 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 8 election: Requested vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:20.070111 25679 tablet_loader.cc:96] loaded metadata for tablet 332aa2d8b2044acf8f787335c4b21449 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:20.071839 25679 tablet_loader.cc:96] loaded metadata for tablet 7a63450281bb4762bdac4271b4a92365 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:20.073371 25679 tablet_loader.cc:96] loaded metadata for tablet 94030938de604fca93313420eff207f4 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:20.076056 25679 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:20.083384 25679 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:20.083767 25679 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:20.088788 25667 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:20.089329 25667 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:20.091648 25667 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:20.093757 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:20.095000 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:20.095763 25260 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 3.
I20260129 03:00:20.097538 25304 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters:
I20260129 03:00:20.098483 25667 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 03:00:20.098861 25667 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:20.099061 25679 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:20.099179 25667 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:20.106498 25679 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Loaded TSK: 0
I20260129 03:00:20.107638 25679 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:20.119041 25053 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 8 FOLLOWER]: Refusing update from remote peer 323384262be54b1395f4569b64e04543: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 8 index: 19. (index mismatch)
I20260129 03:00:20.121023 25630 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260129 03:00:20.124819 25667 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:20.127395 25667 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 4 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:20.128379 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:20.128988 25260 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:20.116815 25589 catalog_manager.cc:5654] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:20.131907 25590 catalog_manager.cc:5654] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e reported cstate change: term changed from 2 to 3, leader changed from 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66) to 9920f2741247430eaf2280bba934199e (127.22.237.65). New cstate: current_term: 3 leader_uuid: "9920f2741247430eaf2280bba934199e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:20.136277 25260 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 4.
I20260129 03:00:20.137372 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:20.137639 25304 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters:
I20260129 03:00:20.138772 25667 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Leader election won for term 4
I20260129 03:00:20.144222 25467 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323384262be54b1395f4569b64e04543. Latest consensus state: current_term: 8 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:20.145102 25467 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:20.145941 25511 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 8 FOLLOWER]: Refusing update from remote peer 323384262be54b1395f4569b64e04543: Log matching property violated. Preceding OpId in replica: term: 6 index: 18. Preceding OpId from leader: term: 8 index: 19. (index mismatch)
I20260129 03:00:20.148794 25630 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260129 03:00:20.155670 25260 raft_consensus.cc:1240] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Rejecting Update request from peer 9920f2741247430eaf2280bba934199e for earlier term 3. Current term is 4. Ops: []
I20260129 03:00:20.157810 25672 consensus_queue.cc:1059] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:20.158478 25667 raft_consensus.cc:697] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 4 LEADER]: Becoming Leader. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:20.158780 25672 raft_consensus.cc:3055] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 3 LEADER]: Stepping down as leader of term 3
I20260129 03:00:20.159174 25672 raft_consensus.cc:740] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9920f2741247430eaf2280bba934199e, State: Running, Role: LEADER
I20260129 03:00:20.159508 25667 consensus_queue.cc:237] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:20.159889 25672 consensus_queue.cc:260] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:20.161870 25672 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:20.194698 25467 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:20.195493 25467 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:20.197602 25630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:20.198603 25630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:20.198223 25698 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:20.199788 25698 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:20.200013 25680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323384262be54b1395f4569b64e04543. Latest consensus state: current_term: 8 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:20.202697 25680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:20.210151 25671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:20.211066 25671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:20.224391 25588 catalog_manager.cc:5654] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 reported cstate change: term changed from 3 to 4, leader changed from 9920f2741247430eaf2280bba934199e (127.22.237.65) to 4c56cc69229649979295105b5d477420 (127.22.237.67). New cstate: current_term: 4 leader_uuid: "4c56cc69229649979295105b5d477420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:20.227809 25680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:20.228621 25680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:20.228749 25208 heartbeater.cc:499] Master 127.22.237.125:43159 was elected leader, sending a full tablet report...
I20260129 03:00:20.242653 25702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "323384262be54b1395f4569b64e04543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:20.243494 25702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:20.246441 25260 raft_consensus.cc:1275] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Refusing update from remote peer 4c56cc69229649979295105b5d477420: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260129 03:00:20.247905 25667 consensus_queue.cc:1048] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:20.252892 25286 heartbeater.cc:499] Master 127.22.237.125:43159 was elected leader, sending a full tablet report...
I20260129 03:00:20.271692 25180 raft_consensus.cc:1275] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Refusing update from remote peer 4c56cc69229649979295105b5d477420: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260129 03:00:20.271895 25364 heartbeater.cc:499] Master 127.22.237.125:43159 was elected leader, sending a full tablet report...
I20260129 03:00:20.272131 25180 pending_rounds.cc:85] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e: Aborting all ops after (but not including) 4
I20260129 03:00:20.272392 25180 pending_rounds.cc:107] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260129 03:00:20.272688 25180 raft_consensus.cc:2889] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260129 03:00:20.277197 25693 consensus_queue.cc:1048] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260129 03:00:20.531735 25281 debug-util.cc:398] Leaking SignalData structure 0x7b08001d1060 after lost signal to thread 23480
W20260129 03:00:20.532995 25281 debug-util.cc:398] Leaking SignalData structure 0x7b08001d17c0 after lost signal to thread 25077
W20260129 03:00:20.534435 25281 debug-util.cc:398] Leaking SignalData structure 0x7b0800173480 after lost signal to thread 25206
W20260129 03:00:20.534762 25281 debug-util.cc:398] Leaking SignalData structure 0x7b0800043b20 after lost signal to thread 25207
W20260129 03:00:20.535971 25281 debug-util.cc:398] Leaking SignalData structure 0x7b08000acb00 after lost signal to thread 25284
W20260129 03:00:20.536278 25281 debug-util.cc:398] Leaking SignalData structure 0x7b0800069e40 after lost signal to thread 25285
W20260129 03:00:20.537320 25281 debug-util.cc:398] Leaking SignalData structure 0x7b080016aca0 after lost signal to thread 25362
W20260129 03:00:20.537561 25281 debug-util.cc:398] Leaking SignalData structure 0x7b080016ac80 after lost signal to thread 25363
W20260129 03:00:20.538396 25281 debug-util.cc:398] Leaking SignalData structure 0x7b08001838a0 after lost signal to thread 25535
W20260129 03:00:20.539235 25281 debug-util.cc:398] Leaking SignalData structure 0x7b0800177ca0 after lost signal to thread 25624
I20260129 03:00:20.703801 23477 master.cc:561] Master@127.22.237.125:43159 shutting down...
I20260129 03:00:20.764895 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 LEADER]: Raft consensus shutting down.
I20260129 03:00:20.766032 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:20.766492 23477 tablet_replica.cc:333] stopping tablet replica
I20260129 03:00:20.790153 23477 master.cc:583] Master@127.22.237.125:43159 shutdown complete.
I20260129 03:00:20.900337 25712 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 323384262be54b1395f4569b64e04543)
I20260129 03:00:20.900933 25712 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:20.905138 25712 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:20.907369 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 9 candidate_status { last_received { term: 8 index: 22 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
W20260129 03:00:20.911520 25486 leader_election.cc:336] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111)
I20260129 03:00:20.912014 25486 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7; no voters: 323384262be54b1395f4569b64e04543, ee59203227af44a69626ad639b3960b6
I20260129 03:00:20.912823 25712 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260129 03:00:20.937951 25715 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 323384262be54b1395f4569b64e04543)
I20260129 03:00:20.938524 25715 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:20.943173 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 9 candidate_status { last_received { term: 8 index: 22 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:20.943836 25511 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee59203227af44a69626ad639b3960b6 in term 8.
I20260129 03:00:20.943014 25715 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:20.945107 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:20.945957 25715 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:20.946689 25715 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 323384262be54b1395f4569b64e04543)
I20260129 03:00:20.947052 25715 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:20.955643 25715 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:20.960879 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 9 candidate_status { last_received { term: 8 index: 22 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
I20260129 03:00:20.961488 25511 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 8 FOLLOWER]: Advancing to term 9
W20260129 03:00:20.965399 25029 leader_election.cc:336] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111)
I20260129 03:00:20.969044 25715 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 9 election: Requested vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:20.969347 25511 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee59203227af44a69626ad639b3960b6 in term 9.
I20260129 03:00:20.970805 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:20.971704 25715 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 9 FOLLOWER]: Leader election won for term 9
W20260129 03:00:20.974612 25029 leader_election.cc:336] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111)
I20260129 03:00:20.976043 25715 raft_consensus.cc:697] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 9 LEADER]: Becoming Leader. State: Replica: ee59203227af44a69626ad639b3960b6, State: Running, Role: LEADER
I20260129 03:00:20.977149 25715 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 8.22, Last appended by leader: 22, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:20.980582 25718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee59203227af44a69626ad639b3960b6. Latest consensus state: current_term: 9 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:20.981325 25718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:20.983189 25719 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:20.990218 25719 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=613936357d19474fb00a3ccc9328458a]
I20260129 03:00:20.997505 25719 tablet_loader.cc:96] loaded metadata for tablet 332aa2d8b2044acf8f787335c4b21449 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:20.998944 25719 tablet_loader.cc:96] loaded metadata for tablet 7a63450281bb4762bdac4271b4a92365 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:21.000339 25719 tablet_loader.cc:96] loaded metadata for tablet 94030938de604fca93313420eff207f4 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:21.003330 25719 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:21.008561 25719 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:21.008869 25719 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:21.013463 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260129 03:00:21.015170 25719 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:21.020740 25719 catalog_manager.cc:6038] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6: Loaded TSK: 0
I20260129 03:00:21.021858 25719 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260129 03:00:21.029527 25720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:21.034076 25721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:21.045629 25723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:21.056978 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:21.058503 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:21.058777 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:21.058983 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655621058963 us; error 0 us; skew 500 ppm
I20260129 03:00:21.059682 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:21.064651 25511 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 9 FOLLOWER]: Refusing update from remote peer ee59203227af44a69626ad639b3960b6: Log matching property violated. Preceding OpId in replica: term: 8 index: 22. Preceding OpId from leader: term: 9 index: 23. (index mismatch)
I20260129 03:00:21.066242 25718 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260129 03:00:21.070365 25712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee59203227af44a69626ad639b3960b6. Latest consensus state: current_term: 9 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:21.071130 25712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:21.082975 25715 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:21.083786 25715 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: LEADER
W20260129 03:00:21.090786 25029 consensus_peers.cc:597] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 -> Peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Couldn't send request to peer 323384262be54b1395f4569b64e04543. Status: Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260129 03:00:21.093313 25712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:21.094100 25712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:21.097160 23477 webserver.cc:492] Webserver started at http://127.22.237.125:42547/ using document root <none> and password file <none>
I20260129 03:00:21.097811 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:21.098047 23477 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260129 03:00:21.102362 23477 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260129 03:00:21.106645 25730 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:21.107873 23477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260129 03:00:21.108274 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root
uuid: "323384262be54b1395f4569b64e04543"
format_stamp: "Formatted at 2026-01-29 03:00:12 on dist-test-slave-1zcf"
I20260129 03:00:21.108800 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:21.131597 25207 heartbeater.cc:499] Master 127.22.237.124:40261 was elected leader, sending a full tablet report...
I20260129 03:00:21.135952 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:21.138271 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:21.164713 25285 heartbeater.cc:499] Master 127.22.237.124:40261 was elected leader, sending a full tablet report...
I20260129 03:00:21.215188 25363 heartbeater.cc:499] Master 127.22.237.124:40261 was elected leader, sending a full tablet report...
I20260129 03:00:21.296950 25781 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.125:43159 every 8 connection(s)
I20260129 03:00:21.296885 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.125:43159
I20260129 03:00:21.319250 25782 sys_catalog.cc:263] Verifying existing consensus state
I20260129 03:00:21.324661 25782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Bootstrap starting.
I20260129 03:00:21.391619 25364 heartbeater.cc:344] Connected to a master server at 127.22.237.125:43159
I20260129 03:00:21.426753 25286 heartbeater.cc:344] Connected to a master server at 127.22.237.125:43159
I20260129 03:00:21.430145 25208 heartbeater.cc:344] Connected to a master server at 127.22.237.125:43159
I20260129 03:00:21.457322 25782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=0 applied=22 ignored=0} inserts{seen=7 ignored=0} mutations{seen=15 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 03:00:21.458387 25782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: Bootstrap complete.
I20260129 03:00:21.461613 25782 raft_consensus.cc:359] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:21.462368 25782 raft_consensus.cc:740] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323384262be54b1395f4569b64e04543, State: Initialized, Role: FOLLOWER
I20260129 03:00:21.463155 25782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 8.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:21.471108 25788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:21.471902 25788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:21.474475 25782 sys_catalog.cc:565] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:21.475062 25757 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:21.533524 25788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee59203227af44a69626ad639b3960b6. Latest consensus state: current_term: 9 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:21.538540 25788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:21.541055 25796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:21.541961 25796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:21.554363 25802 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260129 03:00:21.581645 25718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "ee59203227af44a69626ad639b3960b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:21.582472 25718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:21.602345 25747 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4c56cc69229649979295105b5d477420" instance_seqno: 1769655613686144) as {username='slave'} at 127.0.0.1:56254; Asking this server to re-register.
I20260129 03:00:21.603511 25364 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:21.606453 25747 ts_manager.cc:194] Registered new tserver with Master: 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:21.636519 25747 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" instance_seqno: 1769655613512055) as {username='slave'} at 127.0.0.1:56274; Asking this server to re-register.
I20260129 03:00:21.638478 25286 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:21.646327 25746 ts_manager.cc:194] Registered new tserver with Master: 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:21.648216 25747 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9920f2741247430eaf2280bba934199e" instance_seqno: 1769655613375070) as {username='slave'} at 127.0.0.1:56262; Asking this server to re-register.
I20260129 03:00:21.649583 25208 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:21.656829 25747 ts_manager.cc:194] Registered new tserver with Master: 9920f2741247430eaf2280bba934199e (127.22.237.65:41421)
I20260129 03:00:21.656988 23477 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.342384448s
I20260129 03:00:22.560963 25802 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:22.561305 25802 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: loading cluster ID for follower catalog manager: success
I20260129 03:00:22.568683 25802 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: acquiring CA information for follower catalog manager: success
I20260129 03:00:22.574313 25802 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260129 03:00:33.570202 25827 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ee59203227af44a69626ad639b3960b6)
I20260129 03:00:33.571244 25827 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:33.634377 25828 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ee59203227af44a69626ad639b3960b6)
I20260129 03:00:33.636724 25828 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:33.675599 25828 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:33.677649 25832 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:33.678252 25832 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:33.680922 25832 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:33.683176 25827 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:33.683034 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:33.688690 25835 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:33.689320 25835 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:33.692451 25835 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:33.695263 25836 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:33.695832 25836 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:33.701697 25757 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 10 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:33.702400 25757 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in term 9.
I20260129 03:00:33.703971 25486 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7; no voters:
I20260129 03:00:33.705030 25827 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260129 03:00:33.705518 25827 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ee59203227af44a69626ad639b3960b6)
I20260129 03:00:33.705974 25827 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:33.713382 25837 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:33.715201 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 10 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:33.716004 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:33.698825 25836 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:33.714108 25837 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:33.718572 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:33.719316 25182 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 6.
I20260129 03:00:33.719892 25180 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 2.
I20260129 03:00:33.720739 25302 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:33.720888 25837 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:33.721560 25836 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:33.721942 25836 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:33.722234 25836 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:33.723335 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:33.723968 25337 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 2.
I20260129 03:00:33.725245 25147 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:33.726064 25837 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 03:00:33.726464 25837 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:33.726868 25837 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:33.727759 25836 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:33.728820 25827 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:33.729996 25511 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 323384262be54b1395f4569b64e04543 in current term 10: Already voted for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in this term.
I20260129 03:00:33.731881 25757 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 10 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543"
I20260129 03:00:33.732479 25757 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:33.735236 25837 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:33.738102 25837 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 election: Requested vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:33.738907 25257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:33.740451 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 10 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:33.742101 25337 raft_consensus.cc:1240] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Rejecting Update request from peer 2b1ed11ca9aa4d0e9474478934d90f3b for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:33.742754 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:33.745209 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:33.745885 25182 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 4.
I20260129 03:00:33.746556 25147 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:33.747964 25224 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters: 4c56cc69229649979295105b5d477420
I20260129 03:00:33.750445 25178 raft_consensus.cc:1240] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Rejecting Update request from peer 2b1ed11ca9aa4d0e9474478934d90f3b for earlier term 2. Current term is 3. Ops: []
I20260129 03:00:33.750921 25302 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:33.751782 25835 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260129 03:00:33.745121 25832 consensus_queue.cc:1059] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 03:00:33.752135 25835 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:33.752540 25835 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:33.752585 25837 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 03:00:33.753538 25840 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:33.753916 25840 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:33.754190 25840 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:33.768211 25258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:33.768913 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 10 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6"
I20260129 03:00:33.777433 25843 raft_consensus.cc:3055] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:33.777956 25843 raft_consensus.cc:740] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:33.773093 25827 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 10 election: Requested vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:33.780751 25757 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in term 10.
I20260129 03:00:33.796190 25840 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:33.797369 25258 raft_consensus.cc:1240] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Rejecting Update request from peer 4c56cc69229649979295105b5d477420 for earlier term 4. Current term is 5. Ops: []
I20260129 03:00:33.797847 25756 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Rejecting Update request from peer ee59203227af44a69626ad639b3960b6 for earlier term 9. Current term is 10. Ops: []
I20260129 03:00:33.798674 25840 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 5 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:33.798776 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:33.799389 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:33.781422 25835 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:33.807582 25180 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:33.804364 25843 consensus_queue.cc:260] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:33.808279 25180 raft_consensus.cc:2393] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c56cc69229649979295105b5d477420 in current term 3: Already voted for candidate 9920f2741247430eaf2280bba934199e in this term.
I20260129 03:00:33.809053 25843 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:33.809898 25259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:33.805330 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:33.806458 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:33.811990 25842 consensus_queue.cc:1059] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 03:00:33.805337 25836 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 7 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:33.813025 25842 raft_consensus.cc:3055] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 4 LEADER]: Stepping down as leader of term 4
I20260129 03:00:33.813373 25842 raft_consensus.cc:740] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:33.813763 25225 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b; no voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e
I20260129 03:00:33.806458 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:33.807125 25835 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:33.814903 25840 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260129 03:00:33.814654 25842 consensus_queue.cc:260] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:33.815857 25302 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:33.816093 25842 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:33.822721 25840 consensus_queue.cc:1059] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:33.823792 25832 raft_consensus.cc:3055] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 LEADER]: Stepping down as leader of term 2
I20260129 03:00:33.824389 25832 raft_consensus.cc:740] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
W20260129 03:00:33.824887 25224 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:33.825736 25832 consensus_queue.cc:260] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:33.826759 25844 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260129 03:00:33.827132 25832 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 2 FOLLOWER]: Advancing to term 3
I20260129 03:00:33.836220 25260 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 7.
I20260129 03:00:33.861115 25256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:33.884995 25511 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 10 FOLLOWER]: Rejecting Update request from peer ee59203227af44a69626ad639b3960b6 for earlier term 9. Current term is 10. Ops: []
W20260129 03:00:33.978741 25224 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:34.005039 25734 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.22.237.126:40691 timed out after 0.278s (SENT) [suppressed 20 similar messages]
I20260129 03:00:34.073036 25843 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:34.073714 25843 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:34.075879 25843 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:34.076417 25843 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:34.078279 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:34.079000 25336 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 3.
I20260129 03:00:34.079747 25840 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:34.081468 25225 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters:
I20260129 03:00:34.080235 25840 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:34.083721 25840 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:34.084252 25840 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260129 03:00:34.084630 25840 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:34.084942 25840 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 3 FOLLOWER]: Advancing to term 4
I20260129 03:00:34.076858 25843 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:34.087957 25843 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:34.090246 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:34.091006 25337 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 5.
I20260129 03:00:34.090283 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:34.092386 25336 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 7.
I20260129 03:00:34.093526 25225 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters:
I20260129 03:00:34.094583 25225 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters:
W20260129 03:00:34.102682 25224 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:34.103371 25843 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:34.103808 25843 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:34.104146 25843 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Advancing to term 6
W20260129 03:00:34.145695 25304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.66:43759, user_credentials={real_user=slave}} blocked reactor thread for 49428.4us
I20260129 03:00:34.147531 25256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:34.149338 25304 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:34.150182 25844 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:34.150689 25844 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:34.153146 25844 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:34.153730 25844 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260129 03:00:34.166018 25486 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.125:43159, user_credentials={real_user=slave}} blocked reactor thread for 69301.9us
I20260129 03:00:34.125299 25840 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:34.169377 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:34.169332 25756 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 10 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543"
I20260129 03:00:34.170421 25756 raft_consensus.cc:2376] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in term 10. Re-sending same reply.
I20260129 03:00:34.170332 25336 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 3 FOLLOWER]: Advancing to term 4
W20260129 03:00:34.199791 25304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.66:43759, user_credentials={real_user=slave}} blocked reactor thread for 50515.5us
I20260129 03:00:34.202662 25256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:34.169553 25840 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 4 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:34.211593 25840 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260129 03:00:34.211975 25840 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:34.212378 25224 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:34.212853 25840 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Advancing to term 8
I20260129 03:00:34.248721 25486 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7; no voters: ee59203227af44a69626ad639b3960b6
I20260129 03:00:34.249802 25827 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 10 FOLLOWER]: Leader election won for term 10
I20260129 03:00:34.245939 25843 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:34.395277 25256 raft_consensus.cc:2393] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c56cc69229649979295105b5d477420 in current term 6: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
I20260129 03:00:34.397154 25843 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:34.478063 25827 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 10 LEADER]: Becoming Leader. State: Replica: 8d6b9d190288498a9d76e5abc3e2eff7, State: Running, Role: LEADER
I20260129 03:00:34.479228 25827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 9.23, Last appended by leader: 23, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:34.489622 25851 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:34.490276 25851 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:34.492358 25851 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:34.492877 25851 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:34.468822 25840 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:34.617184 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:34.619105 25181 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:34.546738 25840 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 8 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:34.618244 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:34.675279 25177 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:34.675112 25182 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 4.
I20260129 03:00:34.677799 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:34.678370 25182 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Advancing to term 8
I20260129 03:00:34.493284 25851 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:34.706284 25177 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:34.710354 25179 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 3.
I20260129 03:00:34.709625 25053 raft_consensus.cc:3055] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 9 LEADER]: Stepping down as leader of term 9
I20260129 03:00:34.711856 25053 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ee59203227af44a69626ad639b3960b6, State: Running, Role: LEADER
I20260129 03:00:34.712888 25177 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:34.713009 25053 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 9.23, Last appended by leader: 23, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:34.716542 25053 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:34.716866 25182 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 8.
I20260129 03:00:34.721606 25850 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:34.722298 25851 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
W20260129 03:00:34.722852 25030 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.237.126:40691, user_credentials={real_user=slave}} blocked reactor thread for 486840us
I20260129 03:00:34.722187 25850 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:34.725191 25177 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:34.725818 25177 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 4.
I20260129 03:00:34.727188 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:34.728317 25224 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:34.728612 25848 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6b9d190288498a9d76e5abc3e2eff7. Latest consensus state: current_term: 10 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:34.729100 25840 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 8 FOLLOWER]: Leader election won for term 8
I20260129 03:00:34.729414 25848 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:34.729590 25840 raft_consensus.cc:697] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 8 LEADER]: Becoming Leader. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:34.730556 25840 consensus_queue.cc:237] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
W20260129 03:00:34.778158 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:34.891674 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:34.969643 25148 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.66:43759, user_credentials={real_user=slave}} blocked reactor thread for 216413us
W20260129 03:00:34.995781 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:35.110150 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:35.187050 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:35.228385 25147 leader_election.cc:336] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4c56cc69229649979295105b5d477420 (127.22.237.67:38301): Timed out: RequestConsensusVote RPC to 127.22.237.67:38301 timed out after 0.258s (SENT)
W20260129 03:00:35.229099 25148 leader_election.cc:336] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759): Timed out: RequestConsensusVote RPC to 127.22.237.66:43759 timed out after 0.258s (SENT)
I20260129 03:00:35.229573 25148 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
W20260129 03:00:35.281071 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:35.378822 25848 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:35.379475 25848 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:35.382050 25848 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
W20260129 03:00:35.387197 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:35.427429 25851 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260129 03:00:35.466324 25148 leader_election.cc:336] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759): Timed out: RequestConsensusVote RPC to 127.22.237.66:43759 timed out after 0.372s (SENT)
W20260129 03:00:35.473281 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:35.503739 25147 leader_election.cc:336] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4c56cc69229649979295105b5d477420 (127.22.237.67:38301): Timed out: RequestConsensusVote RPC to 127.22.237.67:38301 timed out after 0.372s (SENT)
I20260129 03:00:35.504269 25147 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:35.505420 25851 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260129 03:00:35.577389 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:35.580129 25041 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45774 (request call id 182) took 1352 ms. Trace:
I20260129 03:00:35.580639 25041 rpcz_store.cc:276] 0129 03:00:34.227224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:35.578686 (+1351462us) service_pool.cc:225] Handling call
0129 03:00:35.579957 (+ 1271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:35.672374 25851 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:35.673002 25851 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:35.675906 25851 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
W20260129 03:00:35.683531 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:35.699086 25042 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45772 (request call id 183) took 1471 ms. Trace:
I20260129 03:00:35.699591 25042 rpcz_store.cc:276] 0129 03:00:34.227585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:35.697441 (+1469856us) service_pool.cc:225] Handling call
0129 03:00:35.698906 (+ 1465us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:35.708384 25302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.65:41421, user_credentials={real_user=slave}} blocked reactor thread for 198744us
W20260129 03:00:35.718397 25029 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.125:43159, user_credentials={real_user=slave}} blocked reactor thread for 130191us
I20260129 03:00:35.720702 25851 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:35.721153 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:35.721237 25851 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:35.721953 25182 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 4.
I20260129 03:00:35.723218 25302 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:34.349296 25336 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 4.
I20260129 03:00:35.724514 25862 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:35.724568 25861 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:35.724979 25862 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:35.725368 25862 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:35.725214 25861 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:35.726918 25043 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:45764 (request call id 185) took 1820 ms. Trace:
I20260129 03:00:35.727298 25851 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
W20260129 03:00:35.724655 25336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47450 (request call id 364) took 1555 ms (client timeout 250 ms). Trace:
I20260129 03:00:35.727299 25043 rpcz_store.cc:276] 0129 03:00:33.906002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:34.712775 (+806773us) service_pool.cc:225] Handling call
0129 03:00:35.725974 (+1013199us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b5400141c54. stack: 00007fb19a297373 00007fb19a297270 00007fb199be9574 00000000003bff74 00000000003bff09 00007fb1a0c3c0c0 00007fb1a0cdabba 00007fb1a30f8eac 00007fb1a31f310e 00007fb1a16f24a4 00007fb1a16f2432 00007fb1a16f2361 00007fb1a16f22dc 00007fb1a16f1592 00007fb19eea934c 00007fb19eea8786
0129 03:00:35.726745 (+ 771us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34322176}
W20260129 03:00:35.728233 25336 rpcz_store.cc:269] 0129 03:00:34.168901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:34.169064 (+ 163us) service_pool.cc:225] Handling call
0129 03:00:35.724557 (+1555493us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260129 03:00:35.728744 25733 leader_election.cc:336] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261): Timed out: RequestConsensusVote RPC to 127.22.237.124:40261 timed out after 0.278s (SENT)
I20260129 03:00:35.731992 25862 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:35.733986 25862 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 6 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:35.734715 25182 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:35.735251 25182 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 4 FOLLOWER]: Advancing to term 6
W20260129 03:00:35.736579 25052 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51498 (request call id 1) took 1012 ms (client timeout 278 ms). Trace:
W20260129 03:00:35.736888 25052 rpcz_store.cc:269] 0129 03:00:34.723730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:35.736281 (+1012551us) service_pool.cc:212] Skipping call since client already timed out
0129 03:00:35.736412 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260129 03:00:35.737412 25179 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33718 (request call id 362) took 1120 ms (client timeout 230 ms). Trace:
W20260129 03:00:35.737931 25179 rpcz_store.cc:269] 0129 03:00:34.616790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:34.616922 (+ 132us) service_pool.cc:225] Handling call
0129 03:00:35.737221 (+1120299us) spinlock_profiling.cc:243] Waited 36.7 ms on lock 0x7b54003502d4. stack: 00007fb19a297373 00007fb19a297270 00007fb199be9574 00000000003bff74 00000000003bff09 00007fb1a0c3b5a1 00007fb1a0cd5f7a 00007fb1a2cd7647 00007fb1a077f834 00007fb1a077f7c2 00007fb1a077f6f1 00007fb1a077f66c 00007fb1a077e922 00007fb19eea934c 00007fb19eea8786 00007fb19eea812f
0129 03:00:35.737323 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":84322048}
I20260129 03:00:35.741695 25182 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 6.
I20260129 03:00:35.742964 25302 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:35.743731 25862 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Leader election won for term 6
I20260129 03:00:35.744745 25862 raft_consensus.cc:697] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 6 LEADER]: Becoming Leader. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:35.745651 25862 consensus_queue.cc:237] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
W20260129 03:00:35.766708 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:35.835753 25182 raft_consensus.cc:1275] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Refusing update from remote peer 4c56cc69229649979295105b5d477420: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260129 03:00:35.837704 25862 consensus_queue.cc:1048] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260129 03:00:35.863895 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:35.922634 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:35.923568 25179 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 3.
I20260129 03:00:35.924840 25302 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:35.925719 25862 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260129 03:00:35.926106 25862 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260129 03:00:35.926110 25029 leader_election.cc:336] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Timed out: RequestConsensusVote RPC to 127.22.237.125:43159 timed out after 0.205s (SENT)
I20260129 03:00:35.926750 25862 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 4 FOLLOWER]: Advancing to term 5
W20260129 03:00:35.928442 25030 leader_election.cc:336] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691): Timed out: RequestConsensusVote RPC to 127.22.237.126:40691 timed out after 0.205s (SENT)
I20260129 03:00:35.928887 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7
W20260129 03:00:35.965046 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:36.076180 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:35.926060 25861 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
W20260129 03:00:36.184326 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:36.200076 25030 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.126:40691, user_credentials={real_user=slave}} blocked reactor thread for 271660us
I20260129 03:00:36.177018 25862 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:36.225102 25862 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 5 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:36.226079 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:36.226752 25179 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 3 FOLLOWER]: Advancing to term 5
I20260129 03:00:36.234112 25179 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 5.
I20260129 03:00:36.235437 25302 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:36.200117 25866 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260129 03:00:36.246260 25863 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 5 FOLLOWER]: Leader election won for term 5
I20260129 03:00:36.246874 25863 raft_consensus.cc:697] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 5 LEADER]: Becoming Leader. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:36.247841 25863 consensus_queue.cc:237] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
W20260129 03:00:36.285575 25147 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.67:38301, user_credentials={real_user=slave}} blocked reactor thread for 257652us
W20260129 03:00:36.304549 25028 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:36.369722 25757 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:36.384317 25179 raft_consensus.cc:1275] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Refusing update from remote peer 4c56cc69229649979295105b5d477420: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260129 03:00:36.385866 25863 consensus_queue.cc:1048] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:36.406925 25258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.407270 25257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.407811 25255 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.408438 25253 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.408901 25254 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.408670 25255 raft_consensus.cc:2393] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9920f2741247430eaf2280bba934199e in current term 4: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
W20260129 03:00:36.429574 25180 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33718 (request call id 363) took 1812 ms (client timeout 288 ms). Trace:
W20260129 03:00:36.430176 25180 rpcz_store.cc:269] 0129 03:00:34.617491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:36.429323 (+1811832us) service_pool.cc:212] Skipping call since client already timed out
0129 03:00:36.429484 (+ 161us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 03:00:36.435374 25849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6b9d190288498a9d76e5abc3e2eff7. Latest consensus state: current_term: 10 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:36.436359 25849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:36.449191 25866 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:36.449306 25259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.449818 25866 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:36.450786 25510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:36.452648 25866 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
W20260129 03:00:36.471467 25225 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.67:38301, user_credentials={real_user=slave}} blocked reactor thread for 61298.4us
I20260129 03:00:36.471758 25757 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:36.477707 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:36.478413 25252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.482170 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
W20260129 03:00:36.482695 25260 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34478 (request call id 178) took 2676 ms (client timeout 290 ms). Trace:
I20260129 03:00:36.482815 25337 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Advancing to term 8
W20260129 03:00:36.490636 25260 rpcz_store.cc:269] 0129 03:00:33.806044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:33.806178 (+ 134us) service_pool.cc:225] Handling call
0129 03:00:36.482509 (+2676331us) spinlock_profiling.cc:243] Waited 8.31 ms on lock 0x7b5400380254. stack: 00007fb19a297373 00007fb19a297270 00007fb199be9574 00000000003bff74 00000000003bff09 00007fb1a0c3b5a1 00007fb1a0cd5f7a 00007fb1a2cd7647 00007fb1a077f834 00007fb1a077f7c2 00007fb1a077f6f1 00007fb1a077f66c 00007fb1a077e922 00007fb19eea934c 00007fb19eea8786 00007fb19eea812f
0129 03:00:36.482612 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19117824}
I20260129 03:00:36.483990 25225 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b; no voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e
I20260129 03:00:36.493934 25874 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260129 03:00:36.485769 25510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:36.494825 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:36.496917 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
W20260129 03:00:36.497335 25734 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 2.491s user 0.025s sys 0.021s
W20260129 03:00:36.497835 25734 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.492s user 0.027s sys 0.021s
W20260129 03:00:36.498138 25734 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.126:40691, user_credentials={real_user=slave}} blocked reactor thread for 2.49304e+06us
I20260129 03:00:36.498353 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:36.498987 25850 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4c56cc69229649979295105b5d477420 (127.22.237.67:38301), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:36.499411 25866 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260129 03:00:36.500240 25260 raft_consensus.cc:2393] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9920f2741247430eaf2280bba934199e in current term 4: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
I20260129 03:00:36.501312 25252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.503144 25225 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b; no voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e
I20260129 03:00:36.505506 25500 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38394 (request call id 145) took 2686 ms. Trace:
I20260129 03:00:36.505888 25500 rpcz_store.cc:276] 0129 03:00:33.818044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:34.277142 (+459098us) service_pool.cc:225] Handling call
0129 03:00:36.504021 (+2226879us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b5400101654. stack: 00007fb19a297373 00007fb19a297270 00007fb199be9574 00000000003bff74 00000000003bff09 00007fb1a0c3c0c0 00007fb1a0cdabba 00007fb1a30f8eac 00007fb1a31f310e 00007fb1a16f24a4 00007fb1a16f2432 00007fb1a16f2361 00007fb1a16f22dc 00007fb1a16f1592 00007fb19eea934c 00007fb19eea8786
0129 03:00:36.504413 (+ 392us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5119177984}
W20260129 03:00:36.509693 25359 debug-util.cc:398] Leaking SignalData structure 0x7b0800211ac0 after lost signal to thread 23480
I20260129 03:00:36.510594 25874 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260129 03:00:36.510879 25359 debug-util.cc:398] Leaking SignalData structure 0x7b0800212220 after lost signal to thread 25077
W20260129 03:00:36.512109 25359 debug-util.cc:398] Leaking SignalData structure 0x7b080011ad80 after lost signal to thread 25206
W20260129 03:00:36.513022 25359 debug-util.cc:398] Leaking SignalData structure 0x7b08001e0cc0 after lost signal to thread 25284
I20260129 03:00:36.513288 25337 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 8.
W20260129 03:00:36.516820 25359 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3b00 after lost signal to thread 25362
W20260129 03:00:36.517120 25359 debug-util.cc:398] Leaking SignalData structure 0x7b08000613e0 after lost signal to thread 25365
W20260129 03:00:36.517907 25359 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3940 after lost signal to thread 25535
I20260129 03:00:36.520742 25501 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38364 (request call id 144) took 2651 ms. Trace:
I20260129 03:00:36.522833 25501 rpcz_store.cc:276] 0129 03:00:33.868793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:34.260899 (+392106us) service_pool.cc:225] Handling call
0129 03:00:36.520165 (+2259266us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b5400101654. stack: 00007fb19a297373 00007fb19a297270 00007fb199be9574 00000000003bff74 00000000003bff09 00007fb1a0c3c0c0 00007fb1a0cdabba 00007fb1a30f8eac 00007fb1a31f310e 00007fb1a16f24a4 00007fb1a16f2432 00007fb1a16f2361 00007fb1a16f22dc 00007fb1a16f1592 00007fb19eea934c 00007fb19eea8786
0129 03:00:36.520567 (+ 402us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4019002624}
W20260129 03:00:36.527259 25359 debug-util.cc:398] Leaking SignalData structure 0x7b0800187b40 after lost signal to thread 25781
W20260129 03:00:36.529124 25256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34478 (request call id 181) took 2328 ms (client timeout 279 ms). Trace:
W20260129 03:00:36.529574 25256 rpcz_store.cc:269] 0129 03:00:34.201124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:34.202290 (+ 1166us) service_pool.cc:225] Handling call
0129 03:00:36.528893 (+2326603us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54003a02d4. stack: 00007fb19a297373 00007fb19a297270 00007fb199be9574 00000000003bff74 00000000003bff09 00007fb1a0c3b5a1 00007fb1a0cd5f7a 00007fb1a2cd7647 00007fb1a077f834 00007fb1a077f7c2 00007fb1a077f6f1 00007fb1a077f66c 00007fb1a077e922 00007fb19eea934c 00007fb19eea8786 00007fb19eea812f
0129 03:00:36.529051 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":440677248}
W20260129 03:00:36.530771 25178 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33710 (request call id 184) took 2036 ms (client timeout 279 ms). Trace:
W20260129 03:00:36.531201 25178 rpcz_store.cc:269] 0129 03:00:34.494544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:36.530550 (+2036006us) service_pool.cc:212] Skipping call since client already timed out
0129 03:00:36.530703 (+ 153us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 03:00:36.522377 25510 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 10 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:36.532711 25747 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:56274 (request call id 114) took 2793 ms. Trace:
I20260129 03:00:36.533370 25734 leader_election.cc:304] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 323384262be54b1395f4569b64e04543; no voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6
I20260129 03:00:36.535414 25747 rpcz_store.cc:276] 0129 03:00:33.738923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:33.739061 (+ 138us) service_pool.cc:225] Handling call
0129 03:00:36.531814 (+2792753us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b5400420254. stack: 00007fb19a297373 00007fb19a297270 00007fb199be9574 00000000003bff74 00000000003bff09 00007fb1a0c3c0c0 00007fb1a0cdabba 00007fb1a30f8eac 00007fb1a31f310e 00007fb1a16f24a4 00007fb1a16f2432 00007fb1a16f2361 00007fb1a16f22dc 00007fb1a16f1592 00007fb19eea934c 00007fb19eea8786
0129 03:00:36.532536 (+ 722us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113131264}
I20260129 03:00:36.537964 25873 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
W20260129 03:00:36.547114 25756 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.866s user 0.018s sys 0.017s
W20260129 03:00:36.547547 25756 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.867s user 0.018s sys 0.018s
I20260129 03:00:36.547691 25755 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
W20260129 03:00:36.551090 25755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56288 (request call id 116) took 181 ms (client timeout 205 ms). Trace:
I20260129 03:00:36.548372 25756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56682 (request call id 3) took 1868 ms. Trace:
I20260129 03:00:36.552268 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
W20260129 03:00:36.551532 25755 rpcz_store.cc:269] 0129 03:00:36.369648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:36.519176 (+149528us) service_pool.cc:225] Handling call
0129 03:00:36.551025 (+ 31849us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:36.557196 25856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6b9d190288498a9d76e5abc3e2eff7. Latest consensus state: current_term: 10 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:36.558905 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:36.561062 25856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:36.561728 25856 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260129 03:00:36.562041 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:36.563366 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:36.556795 25756 rpcz_store.cc:276] 0129 03:00:34.679945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:34.680068 (+ 123us) service_pool.cc:225] Handling call
0129 03:00:34.680317 (+ 249us) raft_consensus.cc:1471] Updating replica for 0 ops
0129 03:00:36.547945 (+1867628us) raft_consensus.cc:1534] Early marking committed up to index 0
0129 03:00:36.547980 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0129 03:00:36.548037 (+ 57us) raft_consensus.cc:1645] Marking committed up to 23
0129 03:00:36.548072 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0129 03:00:36.548161 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
0129 03:00:36.548292 (+ 131us) inbound_call.cc:173] Queueing success response
0129 03:00:36.555936 (+ 7644us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b5400420254. stack: 00007fb19a297373 00007fb19a297270 00007fb199be9574 00000000003bff74 00000000003bff09 00007fb1a0c3c0c0 00007fb1a0cdabba 00007fb1a322b5cc 00007fb1a3233bdc 00007fb1a3233b88 00007fb1a3233b08 00007fb1a3233ac3 00007fb1a3232d6c 00007fb1a11175b6 00007fb1a1117548 00007fb1a0cf121a
Metrics: {"raft.queue_time_us":22934,"raft.run_cpu_time_us":8749,"raft.run_wall_time_us":1857969,"spinlock_wait_cycles":4243534208,"thread_start_us":1867213,"threads_started":1}
I20260129 03:00:36.567282 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:36.567907 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:36.568583 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:36.567881 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:36.570142 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:36.570767 25338 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 8.
I20260129 03:00:36.573058 25256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:36.574009 25252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.574680 25252 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 4.
I20260129 03:00:36.573971 25256 raft_consensus.cc:2393] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c56cc69229649979295105b5d477420 in current term 6: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
I20260129 03:00:36.576232 25260 raft_consensus.cc:1275] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Refusing update from remote peer 4c56cc69229649979295105b5d477420: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260129 03:00:36.577802 25147 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:36.578732 25850 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260129 03:00:36.579386 25147 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:36.580015 25256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:36.580410 25850 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260129 03:00:36.578925 25252 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 4 FOLLOWER]: Advancing to term 5
I20260129 03:00:36.583827 25147 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b
I20260129 03:00:36.584925 25850 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:36.585299 25850 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:36.585613 25850 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:36.588826 25878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:36.590227 25878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:36.592128 25252 raft_consensus.cc:1275] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Refusing update from remote peer 4c56cc69229649979295105b5d477420: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260129 03:00:36.592298 25873 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=613936357d19474fb00a3ccc9328458a]
I20260129 03:00:36.593508 25861 consensus_queue.cc:1048] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:00:36.599920 25256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.600685 25256 raft_consensus.cc:2393] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c56cc69229649979295105b5d477420 in current term 6: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
I20260129 03:00:36.608009 25338 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 8 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260129 03:00:36.609694 25861 consensus_queue.cc:1048] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:36.609699 25874 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260129 03:00:36.612928 25849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:36.613721 25849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:36.615177 25850 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:36.616201 25178 raft_consensus.cc:1240] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Rejecting Update request from peer 2b1ed11ca9aa4d0e9474478934d90f3b for earlier term 8. Current term is 9. Ops: []
W20260129 03:00:36.589203 25734 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.126:40691, user_credentials={real_user=slave}} blocked reactor thread for 55930us
I20260129 03:00:36.629992 25850 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 9 election: Requested vote from peers 4c56cc69229649979295105b5d477420 (127.22.237.67:38301), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:36.632570 25874 consensus_queue.cc:1059] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:36.633093 25256 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:36.633466 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:36.635247 25874 raft_consensus.cc:3055] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 8 LEADER]: Stepping down as leader of term 8
I20260129 03:00:36.635680 25874 raft_consensus.cc:740] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:36.636410 25147 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:36.642432 25874 consensus_queue.cc:260] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:36.645984 25874 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:36.641320 25850 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260129 03:00:36.664798 25873 tablet_loader.cc:96] loaded metadata for tablet 332aa2d8b2044acf8f787335c4b21449 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:36.675812 25873 tablet_loader.cc:96] loaded metadata for tablet 7a63450281bb4762bdac4271b4a92365 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:36.677407 25873 tablet_loader.cc:96] loaded metadata for tablet 94030938de604fca93313420eff207f4 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:36.678917 25873 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:36.679366 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 9 index: 23 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
W20260129 03:00:36.681929 25511 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38414 (request call id 131) took 232 ms (client timeout 205 ms). Trace:
W20260129 03:00:36.682284 25511 rpcz_store.cc:269] 0129 03:00:36.450023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:36.545387 (+ 95364us) service_pool.cc:225] Handling call
0129 03:00:36.681852 (+136465us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260129 03:00:36.688012 25866 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:36.690037 25866 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:36.707410 25873 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:36.707729 25873 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:36.711390 25873 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:36.715441 25873 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: Loaded TSK: 0
I20260129 03:00:36.715226 25877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:36.716019 25877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:36.716794 25873 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:36.763753 25501 catalog_manager.cc:5654] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 reported cstate change: term changed from 2 to 5, leader changed from 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66) to 4c56cc69229649979295105b5d477420 (127.22.237.67). New cstate: current_term: 5 leader_uuid: "4c56cc69229649979295105b5d477420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:36.814254 25862 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:36.814810 25862 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:36.816756 25862 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:36.818312 25178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:36.818392 25252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:36.819916 25252 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 9.
I20260129 03:00:36.821208 25304 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters:
I20260129 03:00:36.822181 25861 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260129 03:00:36.822551 25861 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 2b1ed11ca9aa4d0e9474478934d90f3b)
I20260129 03:00:36.822871 25861 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 8 FOLLOWER]: Advancing to term 9
I20260129 03:00:36.819265 25178 raft_consensus.cc:2393] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c56cc69229649979295105b5d477420 in current term 9: Already voted for candidate 9920f2741247430eaf2280bba934199e in this term.
I20260129 03:00:36.829895 25861 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:36.831951 25861 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 9 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:36.833047 25252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:36.833374 25178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "4c56cc69229649979295105b5d477420" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:36.834218 25178 raft_consensus.cc:2393] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c56cc69229649979295105b5d477420 in current term 9: Already voted for candidate 9920f2741247430eaf2280bba934199e in this term.
I20260129 03:00:36.840418 25252 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c56cc69229649979295105b5d477420 in term 9.
I20260129 03:00:36.841805 25304 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420; no voters: 9920f2741247430eaf2280bba934199e
I20260129 03:00:36.842638 25862 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 9 FOLLOWER]: Leader election won for term 9
I20260129 03:00:36.843156 25862 raft_consensus.cc:697] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 9 LEADER]: Becoming Leader. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:36.844543 25365 heartbeater.cc:499] Master 127.22.237.126:40691 was elected leader, sending a full tablet report...
I20260129 03:00:36.844092 25862 consensus_queue.cc:237] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:36.855381 25501 catalog_manager.cc:5654] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "4c56cc69229649979295105b5d477420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:36.856685 25501 catalog_manager.cc:5654] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 reported cstate change: term changed from 6 to 9, leader changed from 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66) to 4c56cc69229649979295105b5d477420 (127.22.237.67). New cstate: current_term: 9 leader_uuid: "4c56cc69229649979295105b5d477420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:36.900106 25209 heartbeater.cc:499] Master 127.22.237.126:40691 was elected leader, sending a full tablet report...
I20260129 03:00:36.909363 25287 heartbeater.cc:499] Master 127.22.237.126:40691 was elected leader, sending a full tablet report...
I20260129 03:00:36.926931 25178 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Refusing update from remote peer 4c56cc69229649979295105b5d477420: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260129 03:00:36.928385 25862 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:36.942514 25252 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 9 FOLLOWER]: Refusing update from remote peer 4c56cc69229649979295105b5d477420: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260129 03:00:36.944832 25863 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260129 03:00:37.550028 25778 debug-util.cc:398] Leaking SignalData structure 0x7b0800132e20 after lost signal to thread 23480
W20260129 03:00:37.551261 25778 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6040 after lost signal to thread 25077
W20260129 03:00:37.552567 25778 debug-util.cc:398] Leaking SignalData structure 0x7b0800061020 after lost signal to thread 25206
W20260129 03:00:37.554010 25778 debug-util.cc:398] Leaking SignalData structure 0x7b08000fc3c0 after lost signal to thread 25284
W20260129 03:00:37.559625 25778 debug-util.cc:398] Leaking SignalData structure 0x7b08001655e0 after lost signal to thread 25362
W20260129 03:00:37.560115 25778 debug-util.cc:398] Leaking SignalData structure 0x7b0800224ca0 after lost signal to thread 25365
W20260129 03:00:37.562795 25778 debug-util.cc:398] Leaking SignalData structure 0x7b080003cec0 after lost signal to thread 25535
W20260129 03:00:37.563951 25778 debug-util.cc:398] Leaking SignalData structure 0x7b080014de80 after lost signal to thread 25781
W20260129 03:00:37.786600 25304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.237.66:43759, user_credentials={real_user=slave}} blocked reactor thread for 75622.9us
I20260129 03:00:38.425609 25904 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:38.426362 25904 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:38.429281 25904 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:38.443354 25907 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
W20260129 03:00:38.444082 25146 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.007s sys 0.005s
I20260129 03:00:38.444367 25905 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:38.445462 25906 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:38.550971 25906 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:38.553918 25906 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:38.446481 25908 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:38.554735 25908 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:38.557868 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 10 index: 26 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:38.558550 25905 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:38.561313 25905 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4c56cc69229649979295105b5d477420 (127.22.237.67:38301), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:38.564493 25908 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:38.564656 25756 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 10 index: 26 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
W20260129 03:00:38.550035 25146 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.007s sys 0.005s
I20260129 03:00:38.548902 25907 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:38.566594 25029 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:38.568251 25908 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260129 03:00:38.571440 25907 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:38.721598 25909 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:38.722263 25909 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:38.724560 25909 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:38.725271 25903 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:38.725873 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:38.727030 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:38.750464 25903 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:38.751058 25260 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 6.
I20260129 03:00:38.728193 25252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:38.752622 25252 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 9.
I20260129 03:00:38.754325 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:38.754595 25903 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:38.770727 25178 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:38.770862 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e" is_pre_election: true
I20260129 03:00:38.771653 25179 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 5.
I20260129 03:00:38.788059 25224 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b; no voters: 4c56cc69229649979295105b5d477420, 9920f2741247430eaf2280bba934199e
I20260129 03:00:38.789736 25224 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters: 4c56cc69229649979295105b5d477420
I20260129 03:00:38.790482 25903 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260129 03:00:38.791066 25909 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260129 03:00:38.791529 25909 raft_consensus.cc:493] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:38.791942 25909 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:38.802090 25908 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:38.802682 25908 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:38.804603 25908 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:38.806705 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 10 index: 26 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:38.807034 25755 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 10 index: 26 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:38.807760 25755 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee59203227af44a69626ad639b3960b6 in term 10.
I20260129 03:00:38.809010 25029 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 323384262be54b1395f4569b64e04543, ee59203227af44a69626ad639b3960b6; no voters: 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:38.810307 25908 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260129 03:00:38.810710 25908 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:38.811045 25908 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 10 FOLLOWER]: Advancing to term 11
I20260129 03:00:38.832602 25908 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:38.835758 25252 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:38.838073 25148 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:38.887058 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 11 candidate_status { last_received { term: 10 index: 26 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:38.896684 25907 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260129 03:00:38.896589 25148 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.66:43759, user_credentials={real_user=slave}} blocked reactor thread for 58612.1us
I20260129 03:00:38.897317 25907 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:38.897850 25907 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:38.898406 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" is_pre_election: true
I20260129 03:00:38.898984 25260 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 6.
I20260129 03:00:38.900166 25148 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:38.900924 25906 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260129 03:00:38.901269 25906 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4c56cc69229649979295105b5d477420)
I20260129 03:00:38.901595 25906 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:38.892228 25909 raft_consensus.cc:515] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:38.903956 25252 raft_consensus.cc:2393] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9920f2741247430eaf2280bba934199e in current term 6: Already voted for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in this term.
I20260129 03:00:38.904750 25907 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:38.906497 25179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9920f2741247430eaf2280bba934199e"
I20260129 03:00:38.907102 25179 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:38.908385 25906 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:38.910066 25907 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 10 election: Requested vote from peers 4c56cc69229649979295105b5d477420 (127.22.237.67:38301), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
I20260129 03:00:38.910343 25906 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 7 election: Requested vote from peers 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
I20260129 03:00:38.911839 25260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:38.912142 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 10 index: 26 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7"
I20260129 03:00:38.912395 25260 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:38.912896 25756 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 11 candidate_status { last_received { term: 10 index: 26 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543"
I20260129 03:00:38.914492 25029 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:38.918469 25179 raft_consensus.cc:2468] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ed11ca9aa4d0e9474478934d90f3b in term 6.
I20260129 03:00:38.919317 25259 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b"
I20260129 03:00:38.919888 25259 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:38.918469 25260 raft_consensus.cc:2468] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 7.
I20260129 03:00:38.919906 25224 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:38.926911 25903 raft_consensus.cc:2804] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Leader election won for term 6
W20260129 03:00:38.927780 25734 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.020s user 0.015s sys 0.023s
W20260129 03:00:38.928108 25734 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.021s user 0.015s sys 0.023s
I20260129 03:00:38.930389 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 11 candidate_status { last_received { term: 10 index: 26 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
W20260129 03:00:38.931655 25226 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.161s user 0.038s sys 0.037s
W20260129 03:00:38.931984 25226 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.162s user 0.038s sys 0.037s
I20260129 03:00:38.940462 25734 leader_election.cc:304] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 323384262be54b1395f4569b64e04543; no voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6
I20260129 03:00:38.944377 25904 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260129 03:00:38.945590 25259 raft_consensus.cc:2468] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9920f2741247430eaf2280bba934199e in term 10.
I20260129 03:00:38.947453 25254 raft_consensus.cc:1240] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 10 FOLLOWER]: Rejecting Update request from peer 4c56cc69229649979295105b5d477420 for earlier term 9. Current term is 10. Ops: []
I20260129 03:00:38.948006 25148 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
I20260129 03:00:38.948830 25907 raft_consensus.cc:2804] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 7 FOLLOWER]: Leader election won for term 7
I20260129 03:00:38.949296 25907 raft_consensus.cc:697] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 7 LEADER]: Becoming Leader. State: Replica: 9920f2741247430eaf2280bba934199e, State: Running, Role: LEADER
I20260129 03:00:38.950114 25907 consensus_queue.cc:237] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:38.959719 25909 leader_election.cc:290] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [CANDIDATE]: Term 6 election: Requested vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 4c56cc69229649979295105b5d477420 (127.22.237.67:38301)
W20260129 03:00:38.961490 25256 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34470 (request call id 33) took 233 ms (client timeout 203 ms). Trace:
I20260129 03:00:38.961648 25148 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e; no voters:
W20260129 03:00:38.961941 25256 rpcz_store.cc:269] 0129 03:00:38.728078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0129 03:00:38.961263 (+233185us) service_pool.cc:212] Skipping call since client already timed out
0129 03:00:38.961400 (+ 137us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260129 03:00:38.962455 25906 raft_consensus.cc:2804] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 10 FOLLOWER]: Leader election won for term 10
I20260129 03:00:38.962929 25906 raft_consensus.cc:697] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 10 LEADER]: Becoming Leader. State: Replica: 9920f2741247430eaf2280bba934199e, State: Running, Role: LEADER
I20260129 03:00:38.963778 25906 consensus_queue.cc:237] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:38.964181 25908 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 11 election: Requested vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:38.964887 25908 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260129 03:00:38.979327 25499 catalog_manager.cc:5654] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e reported cstate change: term changed from 6 to 7, leader changed from 4c56cc69229649979295105b5d477420 (127.22.237.67) to 9920f2741247430eaf2280bba934199e (127.22.237.65). New cstate: current_term: 7 leader_uuid: "9920f2741247430eaf2280bba934199e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: UNKNOWN } } }
W20260129 03:00:38.983750 25304 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.237.66:43759, user_credentials={real_user=slave}} blocked reactor thread for 64274.9us
I20260129 03:00:38.987371 25338 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:38.988801 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:38.994799 25179 raft_consensus.cc:1240] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 10 LEADER]: Rejecting Update request from peer 4c56cc69229649979295105b5d477420 for earlier term 9. Current term is 10. Ops: []
I20260129 03:00:38.996266 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:39.000366 25337 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:38.993299 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:38.992192 25903 raft_consensus.cc:697] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 LEADER]: Becoming Leader. State: Replica: 2b1ed11ca9aa4d0e9474478934d90f3b, State: Running, Role: LEADER
I20260129 03:00:39.003404 25182 raft_consensus.cc:1240] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Rejecting Update request from peer 4c56cc69229649979295105b5d477420 for earlier term 5. Current term is 6. Ops: []
W20260129 03:00:39.004276 25147 leader_election.cc:336] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4c56cc69229649979295105b5d477420 (127.22.237.67:38301): Timed out: RequestConsensusVote RPC to 127.22.237.67:38301 timed out after 0.203s (SENT)
I20260129 03:00:39.004763 25147 leader_election.cc:304] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9920f2741247430eaf2280bba934199e; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 4c56cc69229649979295105b5d477420
I20260129 03:00:39.007498 25179 raft_consensus.cc:1240] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 7 LEADER]: Rejecting Update request from peer 4c56cc69229649979295105b5d477420 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:39.006742 25906 raft_consensus.cc:2749] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260129 03:00:39.009302 25334 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:39.003496 25335 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94030938de604fca93313420eff207f4" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420" is_pre_election: true
I20260129 03:00:39.010859 25899 consensus_queue.cc:1059] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260129 03:00:39.012033 25333 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a63450281bb4762bdac4271b4a92365" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
I20260129 03:00:39.012008 25336 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "332aa2d8b2044acf8f787335c4b21449" candidate_uuid: "9920f2741247430eaf2280bba934199e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4c56cc69229649979295105b5d477420"
W20260129 03:00:39.005900 25029 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.22.237.125:43159, user_credentials={real_user=slave}} blocked reactor thread for 91499.6us
I20260129 03:00:39.014465 25254 raft_consensus.cc:1240] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Rejecting Update request from peer 4c56cc69229649979295105b5d477420 for earlier term 6. Current term is 7. Ops: []
I20260129 03:00:39.014482 25899 consensus_queue.cc:1243] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [LEADER]: Peer 9920f2741247430eaf2280bba934199e log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260129 03:00:39.016292 25899 raft_consensus.cc:3055] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 9 LEADER]: Stepping down as leader of term 9
I20260129 03:00:39.016554 25899 raft_consensus.cc:740] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:39.017127 25899 consensus_queue.cc:260] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:39.017930 25899 raft_consensus.cc:3060] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 9 FOLLOWER]: Advancing to term 10
I20260129 03:00:39.021131 25916 consensus_queue.cc:1059] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
W20260129 03:00:39.027665 25301 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260129 03:00:39.026211 25903 consensus_queue.cc:237] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:39.022233 25914 consensus_queue.cc:1059] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260129 03:00:39.037590 25052 raft_consensus.cc:1240] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 11 FOLLOWER]: Rejecting Update request from peer 8d6b9d190288498a9d76e5abc3e2eff7 for earlier term 10. Current term is 11. Ops: []
W20260129 03:00:39.023175 25912 consensus_queue.cc:1175] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" responder_term: 10 status { last_received { term: 9 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4c56cc69229649979295105b5d477420 for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260129 03:00:39.040500 25919 raft_consensus.cc:3055] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 5 LEADER]: Stepping down as leader of term 5
I20260129 03:00:39.040390 25878 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 }, Status: INVALID_TERM, Last received: 10.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260129 03:00:39.040493 25914 consensus_queue.cc:1243] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Peer 9920f2741247430eaf2280bba934199e log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260129 03:00:39.043797 25919 raft_consensus.cc:740] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:39.047557 25899 consensus_queue.cc:1059] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
W20260129 03:00:39.049439 25203 debug-util.cc:398] Leaking SignalData structure 0x7b080016b2a0 after lost signal to thread 23480
W20260129 03:00:39.050475 25203 debug-util.cc:398] Leaking SignalData structure 0x7b080015d3a0 after lost signal to thread 25077
W20260129 03:00:39.051666 25203 debug-util.cc:398] Leaking SignalData structure 0x7b0800183380 after lost signal to thread 25206
I20260129 03:00:39.052088 25256 raft_consensus.cc:1240] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 LEADER]: Rejecting Update request from peer 4c56cc69229649979295105b5d477420 for earlier term 5. Current term is 6. Ops: []
W20260129 03:00:39.052765 25203 debug-util.cc:398] Leaking SignalData structure 0x7b080016b7c0 after lost signal to thread 25284
I20260129 03:00:39.049032 25919 consensus_queue.cc:260] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
W20260129 03:00:39.054768 25203 debug-util.cc:398] Leaking SignalData structure 0x7b08001b5020 after lost signal to thread 25362
W20260129 03:00:39.056247 25203 debug-util.cc:398] Leaking SignalData structure 0x7b08000a3820 after lost signal to thread 25535
W20260129 03:00:39.057371 25203 debug-util.cc:398] Leaking SignalData structure 0x7b08001270a0 after lost signal to thread 25781
I20260129 03:00:39.066515 25336 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 10 FOLLOWER]: Refusing update from remote peer 9920f2741247430eaf2280bba934199e: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260129 03:00:39.066816 25256 raft_consensus.cc:1275] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Refusing update from remote peer 9920f2741247430eaf2280bba934199e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260129 03:00:39.068477 25906 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260129 03:00:39.069190 25907 consensus_queue.cc:1048] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260129 03:00:39.075453 25333 raft_consensus.cc:3055] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 6 LEADER]: Stepping down as leader of term 6
I20260129 03:00:39.075906 25333 raft_consensus.cc:740] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4c56cc69229649979295105b5d477420, State: Running, Role: LEADER
I20260129 03:00:39.076731 25333 consensus_queue.cc:260] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:39.078521 25254 raft_consensus.cc:1275] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 10 FOLLOWER]: Refusing update from remote peer 9920f2741247430eaf2280bba934199e: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260129 03:00:39.084364 25303 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260129 03:00:39.084959 25302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.22.237.65:41421, user_credentials={real_user=slave}} blocked reactor thread for 73949.3us
I20260129 03:00:39.084295 25905 consensus_queue.cc:1048] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260129 03:00:39.085713 25333 raft_consensus.cc:3060] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Advancing to term 7
I20260129 03:00:39.093961 25919 raft_consensus.cc:3060] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 5 FOLLOWER]: Advancing to term 6
I20260129 03:00:39.119743 25922 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 10 LEADER]: Stepping down as leader of term 10
I20260129 03:00:39.120183 25922 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 8d6b9d190288498a9d76e5abc3e2eff7, State: Running, Role: LEADER
I20260129 03:00:39.120985 25922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 10.27, Last appended by leader: 27, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
W20260129 03:00:39.123924 23482 debug-util.cc:398] Leaking SignalData structure 0x7b080002b6a0 after lost signal to thread 25226
I20260129 03:00:39.130599 25922 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 10 FOLLOWER]: Advancing to term 11
I20260129 03:00:39.137393 25182 raft_consensus.cc:1275] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260129 03:00:39.139346 25903 consensus_queue.cc:1048] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:39.151129 25333 raft_consensus.cc:1275] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Refusing update from remote peer 9920f2741247430eaf2280bba934199e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260129 03:00:39.171753 25912 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260129 03:00:39.174571 25905 consensus_queue.cc:1048] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260129 03:00:39.182632 25335 raft_consensus.cc:1275] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Refusing update from remote peer 2b1ed11ca9aa4d0e9474478934d90f3b: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260129 03:00:39.184988 25903 consensus_queue.cc:1048] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260129 03:00:39.235916 25911 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:39.236415 25911 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:39.238443 25911 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
I20260129 03:00:39.242275 25756 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 12 candidate_status { last_received { term: 10 index: 26 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:39.244060 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee59203227af44a69626ad639b3960b6" candidate_term: 12 candidate_status { last_received { term: 10 index: 26 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:39.244900 25511 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee59203227af44a69626ad639b3960b6 for term 12 because replica has last-logged OpId of term: 10 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 26.
I20260129 03:00:39.249944 25030 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:39.251631 25911 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260129 03:00:39.283829 25922 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:39.284423 25922 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:39.286450 25922 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:39.289012 25052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 12 candidate_status { last_received { term: 10 index: 27 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:39.289740 25052 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in term 11.
I20260129 03:00:39.290954 25486 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6; no voters:
I20260129 03:00:39.291800 25922 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260129 03:00:39.292179 25922 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:39.292541 25922 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 11 FOLLOWER]: Advancing to term 12
I20260129 03:00:39.292402 25756 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 12 candidate_status { last_received { term: 10 index: 27 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543" is_pre_election: true
I20260129 03:00:39.299539 25922 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:39.301600 25922 leader_election.cc:290] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 12 election: Requested vote from peers 323384262be54b1395f4569b64e04543 (127.22.237.125:43159), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:39.309887 25755 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 12 candidate_status { last_received { term: 10 index: 27 } } ignore_live_leader: false dest_uuid: "323384262be54b1395f4569b64e04543"
I20260129 03:00:39.312180 25052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" candidate_term: 12 candidate_status { last_received { term: 10 index: 27 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6"
I20260129 03:00:39.312814 25052 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 11 FOLLOWER]: Advancing to term 12
I20260129 03:00:39.324776 25904 raft_consensus.cc:493] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:39.325346 25904 raft_consensus.cc:515] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:39.327380 25904 leader_election.cc:290] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), ee59203227af44a69626ad639b3960b6 (127.22.237.124:40261)
I20260129 03:00:39.330525 25053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 11 candidate_status { last_received { term: 10 index: 27 } } ignore_live_leader: false dest_uuid: "ee59203227af44a69626ad639b3960b6" is_pre_election: true
I20260129 03:00:39.330906 25511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "323384262be54b1395f4569b64e04543" candidate_term: 11 candidate_status { last_received { term: 10 index: 27 } } ignore_live_leader: false dest_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" is_pre_election: true
I20260129 03:00:39.331368 25052 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6b9d190288498a9d76e5abc3e2eff7 in term 12.
I20260129 03:00:39.331667 25511 raft_consensus.cc:2368] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 323384262be54b1395f4569b64e04543 for earlier term 11. Current term is 12.
I20260129 03:00:39.332768 25734 leader_election.cc:400] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 323384262be54b1395f4569b64e04543 for earlier term 11. Current term is 12.
I20260129 03:00:39.333076 25734 leader_election.cc:403] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260129 03:00:39.338335 25904 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 10 FOLLOWER]: Advancing to term 12
I20260129 03:00:39.342478 25486 leader_election.cc:304] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d6b9d190288498a9d76e5abc3e2eff7, ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543
I20260129 03:00:39.343376 25922 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 12 FOLLOWER]: Leader election won for term 12
I20260129 03:00:39.343878 25922 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 12 LEADER]: Becoming Leader. State: Replica: 8d6b9d190288498a9d76e5abc3e2eff7, State: Running, Role: LEADER
I20260129 03:00:39.344673 25922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 10.27, Last appended by leader: 27, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:39.347419 25922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6b9d190288498a9d76e5abc3e2eff7. Latest consensus state: current_term: 12 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:39.348093 25922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:39.363746 25904 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 323384262be54b1395f4569b64e04543 for earlier term 11. Current term is 12.
I20260129 03:00:39.426452 25052 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 12 FOLLOWER]: Refusing update from remote peer 8d6b9d190288498a9d76e5abc3e2eff7: Log matching property violated. Preceding OpId in replica: term: 10 index: 26. Preceding OpId from leader: term: 12 index: 28. (index mismatch)
I20260129 03:00:39.428241 25922 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20260129 03:00:39.441192 25756 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 12 FOLLOWER]: Refusing update from remote peer 8d6b9d190288498a9d76e5abc3e2eff7: Log matching property violated. Preceding OpId in replica: term: 10 index: 27. Preceding OpId from leader: term: 12 index: 28. (index mismatch)
I20260129 03:00:39.445196 25922 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.000s
I20260129 03:00:39.462854 25904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6b9d190288498a9d76e5abc3e2eff7. Latest consensus state: current_term: 12 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:39.463618 25904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:39.491766 25911 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6b9d190288498a9d76e5abc3e2eff7. Latest consensus state: current_term: 12 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:39.492550 25911 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:39.507109 25911 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 12 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:39.507856 25911 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:39.516063 25878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 12 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:39.516819 25878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:39.518071 25878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 12 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:39.518810 25878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:39.528237 25904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 12 leader_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } } }
I20260129 03:00:39.528970 25904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [sys.catalog]: This master's current role is: FOLLOWER
I20260129 03:00:39.548214 25935 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:39.555917 25935 catalog_manager.cc:679] Loaded metadata for table NewTableName [id=613936357d19474fb00a3ccc9328458a]
I20260129 03:00:39.563385 25935 tablet_loader.cc:96] loaded metadata for tablet 332aa2d8b2044acf8f787335c4b21449 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:39.564867 25935 tablet_loader.cc:96] loaded metadata for tablet 7a63450281bb4762bdac4271b4a92365 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:39.566203 25935 tablet_loader.cc:96] loaded metadata for tablet 94030938de604fca93313420eff207f4 (table NewTableName [id=613936357d19474fb00a3ccc9328458a])
I20260129 03:00:39.570506 25935 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:39.583621 25935 catalog_manager.cc:1269] Loaded cluster ID: e33a8f87fa4e4d79bda4bbf71d69ec02
I20260129 03:00:39.583945 25935 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:39.589835 25935 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:39.613279 25935 catalog_manager.cc:6038] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7: Loaded TSK: 0
I20260129 03:00:39.622236 25935 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:39.741307 25209 heartbeater.cc:499] Master 127.22.237.126:40691 was elected leader, sending a full tablet report...
I20260129 03:00:39.756305 25287 heartbeater.cc:499] Master 127.22.237.126:40691 was elected leader, sending a full tablet report...
I20260129 03:00:39.761269 25499 catalog_manager.cc:5654] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e reported cstate change: term changed from 9 to 10, leader changed from 4c56cc69229649979295105b5d477420 (127.22.237.67) to 9920f2741247430eaf2280bba934199e (127.22.237.65). New cstate: current_term: 10 leader_uuid: "9920f2741247430eaf2280bba934199e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } health_report { overall_health: HEALTHY } } }
I20260129 03:00:39.763550 25499 catalog_manager.cc:5654] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e reported cstate change: term changed from 5 to 6, leader changed from 4c56cc69229649979295105b5d477420 (127.22.237.67) to 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66). New cstate: current_term: 6 leader_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } }
I20260129 03:00:39.773624 25365 heartbeater.cc:499] Master 127.22.237.126:40691 was elected leader, sending a full tablet report...
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:784: Failure
Expected equality of these values:
live_row_count
Which is: 67
table_info->GetMetrics()->live_row_count->value()
Which is: 134
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 784, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260129 03:00:52.118065 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:00:52.158025 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:52.158874 23477 tablet_replica.cc:333] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e: stopping tablet replica
I20260129 03:00:52.159526 23477 raft_consensus.cc:2243] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 7 LEADER]: Raft consensus shutting down.
I20260129 03:00:52.160740 23477 raft_consensus.cc:2272] T 332aa2d8b2044acf8f787335c4b21449 P 9920f2741247430eaf2280bba934199e [term 7 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.163532 23477 tablet_replica.cc:333] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e: stopping tablet replica
I20260129 03:00:52.164072 23477 raft_consensus.cc:2243] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:52.164588 23477 raft_consensus.cc:2272] T 94030938de604fca93313420eff207f4 P 9920f2741247430eaf2280bba934199e [term 6 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.166592 23477 tablet_replica.cc:333] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e: stopping tablet replica
I20260129 03:00:52.167001 23477 raft_consensus.cc:2243] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 10 LEADER]: Raft consensus shutting down.
I20260129 03:00:52.167781 23477 raft_consensus.cc:2272] T 7a63450281bb4762bdac4271b4a92365 P 9920f2741247430eaf2280bba934199e [term 10 FOLLOWER]: Raft consensus is shut down!
W20260129 03:00:52.186534 25224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.237.65:41421: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20260129 03:00:52.190794 25224 consensus_peers.cc:597] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b -> Peer 9920f2741247430eaf2280bba934199e (127.22.237.65:41421): Couldn't send request to peer 9920f2741247430eaf2280bba934199e. Status: Network error: Client connection negotiation failed: client connection to 127.22.237.65:41421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260129 03:00:52.225057 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:00:52.241282 23477 tablet_server.cc:178] TabletServer@127.22.237.66:0 shutting down...
I20260129 03:00:52.276630 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:52.277289 23477 tablet_replica.cc:333] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b: stopping tablet replica
I20260129 03:00:52.277959 23477 raft_consensus.cc:2243] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 LEADER]: Raft consensus shutting down.
I20260129 03:00:52.278795 23477 raft_consensus.cc:2272] T 94030938de604fca93313420eff207f4 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 6 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.280859 23477 tablet_replica.cc:333] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b: stopping tablet replica
I20260129 03:00:52.281327 23477 raft_consensus.cc:2243] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:52.281749 23477 raft_consensus.cc:2272] T 332aa2d8b2044acf8f787335c4b21449 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 7 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.283496 23477 tablet_replica.cc:333] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b: stopping tablet replica
I20260129 03:00:52.283939 23477 raft_consensus.cc:2243] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 10 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:52.284407 23477 raft_consensus.cc:2272] T 7a63450281bb4762bdac4271b4a92365 P 2b1ed11ca9aa4d0e9474478934d90f3b [term 10 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.302734 25963 raft_consensus.cc:493] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 9920f2741247430eaf2280bba934199e)
I20260129 03:00:52.303220 25963 raft_consensus.cc:515] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } }
I20260129 03:00:52.306208 25963 leader_election.cc:290] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
W20260129 03:00:52.311437 25302 leader_election.cc:336] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 9920f2741247430eaf2280bba934199e (127.22.237.65:41421): Network error: Client connection negotiation failed: client connection to 127.22.237.65:41421: connect: Connection refused (error 111)
W20260129 03:00:52.312075 25304 leader_election.cc:336] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759): Network error: Client connection negotiation failed: client connection to 127.22.237.66:43759: connect: Connection refused (error 111)
I20260129 03:00:52.312412 25304 leader_election.cc:304] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:52.313052 25963 raft_consensus.cc:2749] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260129 03:00:52.331533 23477 tablet_server.cc:195] TabletServer@127.22.237.66:0 shutdown complete.
I20260129 03:00:52.345579 23477 tablet_server.cc:178] TabletServer@127.22.237.67:0 shutting down...
I20260129 03:00:52.348971 25963 raft_consensus.cc:493] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 9920f2741247430eaf2280bba934199e)
I20260129 03:00:52.349464 25963 raft_consensus.cc:515] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920f2741247430eaf2280bba934199e" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41421 } } peers { permanent_uuid: "2b1ed11ca9aa4d0e9474478934d90f3b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43759 } } peers { permanent_uuid: "4c56cc69229649979295105b5d477420" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 38301 } }
I20260129 03:00:52.351042 25963 leader_election.cc:290] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 9920f2741247430eaf2280bba934199e (127.22.237.65:41421), 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759)
W20260129 03:00:52.354462 25304 leader_election.cc:336] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 2b1ed11ca9aa4d0e9474478934d90f3b (127.22.237.66:43759): Network error: Client connection negotiation failed: client connection to 127.22.237.66:43759: connect: Connection refused (error 111)
W20260129 03:00:52.355022 25302 leader_election.cc:336] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 9920f2741247430eaf2280bba934199e (127.22.237.65:41421): Network error: Client connection negotiation failed: client connection to 127.22.237.65:41421: connect: Connection refused (error 111)
I20260129 03:00:52.355314 25302 leader_election.cc:304] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c56cc69229649979295105b5d477420; no voters: 2b1ed11ca9aa4d0e9474478934d90f3b, 9920f2741247430eaf2280bba934199e
I20260129 03:00:52.355873 25963 raft_consensus.cc:2749] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260129 03:00:52.378834 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:52.379491 23477 tablet_replica.cc:333] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420: stopping tablet replica
I20260129 03:00:52.380210 23477 raft_consensus.cc:2243] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 10 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:52.380766 23477 raft_consensus.cc:2272] T 7a63450281bb4762bdac4271b4a92365 P 4c56cc69229649979295105b5d477420 [term 10 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.383293 23477 tablet_replica.cc:333] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420: stopping tablet replica
I20260129 03:00:52.383877 23477 raft_consensus.cc:2243] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:52.384336 23477 raft_consensus.cc:2272] T 94030938de604fca93313420eff207f4 P 4c56cc69229649979295105b5d477420 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.386358 23477 tablet_replica.cc:333] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420: stopping tablet replica
I20260129 03:00:52.386821 23477 raft_consensus.cc:2243] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:52.387217 23477 raft_consensus.cc:2272] T 332aa2d8b2044acf8f787335c4b21449 P 4c56cc69229649979295105b5d477420 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.431459 23477 tablet_server.cc:195] TabletServer@127.22.237.67:0 shutdown complete.
I20260129 03:00:52.446643 23477 master.cc:561] Master@127.22.237.126:40691 shutting down...
I20260129 03:00:52.468864 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 12 LEADER]: Raft consensus shutting down.
I20260129 03:00:52.470263 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8d6b9d190288498a9d76e5abc3e2eff7 [term 12 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.470732 23477 tablet_replica.cc:333] stopping tablet replica
I20260129 03:00:52.491902 23477 master.cc:583] Master@127.22.237.126:40691 shutdown complete.
I20260129 03:00:52.507650 23477 master.cc:561] Master@127.22.237.125:43159 shutting down...
I20260129 03:00:52.528015 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 12 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:52.528678 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 323384262be54b1395f4569b64e04543 [term 12 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.529052 23477 tablet_replica.cc:333] stopping tablet replica
I20260129 03:00:52.573336 23477 master.cc:583] Master@127.22.237.125:43159 shutdown complete.
I20260129 03:00:52.588008 23477 master.cc:561] Master@127.22.237.124:40261 shutting down...
I20260129 03:00:52.593853 25967 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 8d6b9d190288498a9d76e5abc3e2eff7)
I20260129 03:00:52.595084 25967 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d6b9d190288498a9d76e5abc3e2eff7" member_type: VOTER last_known_addr { host: "127.22.237.126" port: 40691 } } peers { permanent_uuid: "323384262be54b1395f4569b64e04543" member_type: VOTER last_known_addr { host: "127.22.237.125" port: 43159 } } peers { permanent_uuid: "ee59203227af44a69626ad639b3960b6" member_type: VOTER last_known_addr { host: "127.22.237.124" port: 40261 } }
I20260129 03:00:52.597982 25967 leader_election.cc:290] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691), 323384262be54b1395f4569b64e04543 (127.22.237.125:43159)
W20260129 03:00:52.608851 25030 leader_election.cc:336] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 8d6b9d190288498a9d76e5abc3e2eff7 (127.22.237.126:40691): Network error: Client connection negotiation failed: client connection to 127.22.237.126:40691: connect: Connection refused (error 111)
W20260129 03:00:52.610298 25029 leader_election.cc:336] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 323384262be54b1395f4569b64e04543 (127.22.237.125:43159): Network error: Client connection negotiation failed: client connection to 127.22.237.125:43159: connect: Connection refused (error 111)
I20260129 03:00:52.610754 25029 leader_election.cc:304] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee59203227af44a69626ad639b3960b6; no voters: 323384262be54b1395f4569b64e04543, 8d6b9d190288498a9d76e5abc3e2eff7
I20260129 03:00:52.611515 25967 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260129 03:00:52.615875 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 12 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:52.616402 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6 [term 12 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:52.616691 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P ee59203227af44a69626ad639b3960b6: stopping tablet replica
I20260129 03:00:52.637039 23477 master.cc:583] Master@127.22.237.124:40261 shutdown complete.
I20260129 03:00:52.662158 23477 test_util.cc:183] -----------------------------------------------
I20260129 03:00:52.662377 23477 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1769655599110862-23477-0
[ FAILED ] TsTabletManagerITest.TestTableStats (39849 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20260129 03:00:52.668685 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:40599
I20260129 03:00:52.669788 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:52.674829 25975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:52.675379 25976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:52.677042 25978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:52.677680 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:52.678516 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:52.678712 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:52.678850 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655652678838 us; error 0 us; skew 500 ppm
I20260129 03:00:52.679358 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:52.681617 23477 webserver.cc:492] Webserver started at http://127.22.237.126:41121/ using document root <none> and password file <none>
I20260129 03:00:52.682125 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:52.682293 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:52.682528 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:52.683651 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "e4f6c06a38fa486b90e5caab3c5bdb45"
format_stamp: "Formatted at 2026-01-29 03:00:52 on dist-test-slave-1zcf"
I20260129 03:00:52.688050 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260129 03:00:52.691361 25983 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:52.692111 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260129 03:00:52.692382 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "e4f6c06a38fa486b90e5caab3c5bdb45"
format_stamp: "Formatted at 2026-01-29 03:00:52 on dist-test-slave-1zcf"
I20260129 03:00:52.692656 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:52.713461 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:52.714711 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:52.752574 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:40599
I20260129 03:00:52.752671 26034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:40599 every 8 connection(s)
I20260129 03:00:52.756395 26035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:52.767395 26035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45: Bootstrap starting.
I20260129 03:00:52.772025 26035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:52.776286 26035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45: No bootstrap required, opened a new log
I20260129 03:00:52.778609 26035 raft_consensus.cc:359] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f6c06a38fa486b90e5caab3c5bdb45" member_type: VOTER }
I20260129 03:00:52.779106 26035 raft_consensus.cc:385] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:52.779362 26035 raft_consensus.cc:740] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f6c06a38fa486b90e5caab3c5bdb45, State: Initialized, Role: FOLLOWER
I20260129 03:00:52.780011 26035 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f6c06a38fa486b90e5caab3c5bdb45" member_type: VOTER }
I20260129 03:00:52.780471 26035 raft_consensus.cc:399] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:52.780720 26035 raft_consensus.cc:493] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:52.781066 26035 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:52.785485 26035 raft_consensus.cc:515] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f6c06a38fa486b90e5caab3c5bdb45" member_type: VOTER }
I20260129 03:00:52.786048 26035 leader_election.cc:304] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4f6c06a38fa486b90e5caab3c5bdb45; no voters:
I20260129 03:00:52.787132 26035 leader_election.cc:290] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:52.787438 26038 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:52.788694 26038 raft_consensus.cc:697] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 1 LEADER]: Becoming Leader. State: Replica: e4f6c06a38fa486b90e5caab3c5bdb45, State: Running, Role: LEADER
I20260129 03:00:52.789402 26038 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f6c06a38fa486b90e5caab3c5bdb45" member_type: VOTER }
I20260129 03:00:52.790019 26035 sys_catalog.cc:565] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:52.792507 26039 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4f6c06a38fa486b90e5caab3c5bdb45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f6c06a38fa486b90e5caab3c5bdb45" member_type: VOTER } }
I20260129 03:00:52.792622 26040 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4f6c06a38fa486b90e5caab3c5bdb45. Latest consensus state: current_term: 1 leader_uuid: "e4f6c06a38fa486b90e5caab3c5bdb45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4f6c06a38fa486b90e5caab3c5bdb45" member_type: VOTER } }
I20260129 03:00:52.793330 26039 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:52.793386 26040 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:52.803103 26044 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:52.808107 26044 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:52.809199 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:52.817247 26044 catalog_manager.cc:1357] Generated new cluster ID: 63c437c1e6624d95a5b45e463ff6f2b9
I20260129 03:00:52.817595 26044 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:52.843178 26044 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:52.844476 26044 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:52.857298 26044 catalog_manager.cc:6027] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45: Generated new TSK 0
I20260129 03:00:52.857937 26044 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:52.876390 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:52.882297 26056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:52.883199 26057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:52.885169 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:52.885537 26059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:52.886620 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:52.886816 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:52.886945 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655652886933 us; error 0 us; skew 500 ppm
I20260129 03:00:52.887418 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:52.889829 23477 webserver.cc:492] Webserver started at http://127.22.237.65:39501/ using document root <none> and password file <none>
I20260129 03:00:52.890301 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:52.890462 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:52.890688 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:52.892067 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "aa3e720de6004bd2893eabcd417658c2"
format_stamp: "Formatted at 2026-01-29 03:00:52 on dist-test-slave-1zcf"
I20260129 03:00:52.896648 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260129 03:00:52.900240 26064 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:52.901002 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 03:00:52.901317 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "aa3e720de6004bd2893eabcd417658c2"
format_stamp: "Formatted at 2026-01-29 03:00:52 on dist-test-slave-1zcf"
I20260129 03:00:52.901618 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:52.914214 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:52.915429 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:52.916944 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:52.919247 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:52.919446 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:52.919700 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:52.919844 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:52.959890 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:32977
I20260129 03:00:52.960005 26126 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:32977 every 8 connection(s)
I20260129 03:00:52.964331 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:52.971987 26131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:52.973058 26132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:52.975579 26134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:52.976001 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:52.977540 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:52.977808 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:52.977964 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655652977949 us; error 0 us; skew 500 ppm
I20260129 03:00:52.978255 26127 heartbeater.cc:344] Connected to a master server at 127.22.237.126:40599
I20260129 03:00:52.978555 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:52.978611 26127 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:52.979449 26127 heartbeater.cc:507] Master 127.22.237.126:40599 requested a full tablet report, sending...
I20260129 03:00:52.981251 23477 webserver.cc:492] Webserver started at http://127.22.237.66:45957/ using document root <none> and password file <none>
I20260129 03:00:52.981982 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:52.981877 26000 ts_manager.cc:194] Registered new tserver with Master: aa3e720de6004bd2893eabcd417658c2 (127.22.237.65:32977)
I20260129 03:00:52.982306 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:52.982648 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:52.984066 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-1-root/instance:
uuid: "1e56b1cc76b147119bf29a3daff53e8b"
format_stamp: "Formatted at 2026-01-29 03:00:52 on dist-test-slave-1zcf"
I20260129 03:00:52.984393 26000 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34010
I20260129 03:00:52.989053 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260129 03:00:52.992759 26139 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:52.993536 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:52.993907 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-1-root
uuid: "1e56b1cc76b147119bf29a3daff53e8b"
format_stamp: "Formatted at 2026-01-29 03:00:52 on dist-test-slave-1zcf"
I20260129 03:00:52.994194 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:53.009881 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:53.011106 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:53.012748 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:53.015041 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:53.015230 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:53.015470 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:53.015625 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:53.057355 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.66:43137
I20260129 03:00:53.057461 26201 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.66:43137 every 8 connection(s)
I20260129 03:00:53.062095 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:53.069973 26205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:53.070228 26206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:53.073521 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:53.073463 26209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:53.073484 26202 heartbeater.cc:344] Connected to a master server at 127.22.237.126:40599
I20260129 03:00:53.074517 26202 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:53.074994 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:53.075225 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:53.075237 26202 heartbeater.cc:507] Master 127.22.237.126:40599 requested a full tablet report, sending...
I20260129 03:00:53.075464 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655653075450 us; error 0 us; skew 500 ppm
I20260129 03:00:53.076102 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:53.077602 26000 ts_manager.cc:194] Registered new tserver with Master: 1e56b1cc76b147119bf29a3daff53e8b (127.22.237.66:43137)
I20260129 03:00:53.078981 23477 webserver.cc:492] Webserver started at http://127.22.237.67:38897/ using document root <none> and password file <none>
I20260129 03:00:53.079484 26000 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34012
I20260129 03:00:53.079687 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:53.079964 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:53.080278 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:53.081498 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-2-root/instance:
uuid: "4c6e67967bb3473881a1721c9c923231"
format_stamp: "Formatted at 2026-01-29 03:00:53 on dist-test-slave-1zcf"
I20260129 03:00:53.086275 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260129 03:00:53.089937 26213 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:53.090765 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260129 03:00:53.091172 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-2-root
uuid: "4c6e67967bb3473881a1721c9c923231"
format_stamp: "Formatted at 2026-01-29 03:00:53 on dist-test-slave-1zcf"
I20260129 03:00:53.091497 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:53.111666 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:53.112938 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:53.114509 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:00:53.116792 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:53.117002 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:53.117220 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:53.117374 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:53.157584 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.67:46355
I20260129 03:00:53.157722 26275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.67:46355 every 8 connection(s)
I20260129 03:00:53.171190 26276 heartbeater.cc:344] Connected to a master server at 127.22.237.126:40599
I20260129 03:00:53.171569 26276 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:53.172441 26276 heartbeater.cc:507] Master 127.22.237.126:40599 requested a full tablet report, sending...
I20260129 03:00:53.174705 26000 ts_manager.cc:194] Registered new tserver with Master: 4c6e67967bb3473881a1721c9c923231 (127.22.237.67:46355)
I20260129 03:00:53.175480 23477 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014703836s
I20260129 03:00:53.176100 26000 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34024
I20260129 03:00:53.212184 26000 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260129 03:00:53.213915 26000 catalog_manager.cc:7016] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260129 03:00:53.255991 26167 tablet_service.cc:1505] Processing CreateTablet for tablet a94827a61f1748dca7119466aa9f0b14 (DEFAULT_TABLE table=test-table [id=22037cf5fedf47b99911bca3a81110d6]), partition=
I20260129 03:00:53.256021 26241 tablet_service.cc:1505] Processing CreateTablet for tablet a94827a61f1748dca7119466aa9f0b14 (DEFAULT_TABLE table=test-table [id=22037cf5fedf47b99911bca3a81110d6]), partition=
I20260129 03:00:53.257131 26167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94827a61f1748dca7119466aa9f0b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:53.257294 26241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94827a61f1748dca7119466aa9f0b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:53.261329 26092 tablet_service.cc:1505] Processing CreateTablet for tablet a94827a61f1748dca7119466aa9f0b14 (DEFAULT_TABLE table=test-table [id=22037cf5fedf47b99911bca3a81110d6]), partition=
I20260129 03:00:53.262553 26092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94827a61f1748dca7119466aa9f0b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:53.272464 26292 tablet_bootstrap.cc:492] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: Bootstrap starting.
I20260129 03:00:53.278223 26292 tablet_bootstrap.cc:654] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:53.281299 26293 tablet_bootstrap.cc:492] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: Bootstrap starting.
I20260129 03:00:53.284462 26294 tablet_bootstrap.cc:492] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Bootstrap starting.
I20260129 03:00:53.286830 26293 tablet_bootstrap.cc:654] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:53.288769 26292 tablet_bootstrap.cc:492] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: No bootstrap required, opened a new log
I20260129 03:00:53.289335 26292 ts_tablet_manager.cc:1403] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260129 03:00:53.291486 26294 tablet_bootstrap.cc:654] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:53.292665 26293 tablet_bootstrap.cc:492] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: No bootstrap required, opened a new log
I20260129 03:00:53.292275 26292 raft_consensus.cc:359] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.293136 26292 raft_consensus.cc:385] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:53.293229 26293 ts_tablet_manager.cc:1403] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260129 03:00:53.293543 26292 raft_consensus.cc:740] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c6e67967bb3473881a1721c9c923231, State: Initialized, Role: FOLLOWER
I20260129 03:00:53.294425 26292 consensus_queue.cc:260] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.296361 26293 raft_consensus.cc:359] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.297160 26293 raft_consensus.cc:385] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:53.297461 26293 raft_consensus.cc:740] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e56b1cc76b147119bf29a3daff53e8b, State: Initialized, Role: FOLLOWER
I20260129 03:00:53.298501 26293 consensus_queue.cc:260] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.299224 26276 heartbeater.cc:499] Master 127.22.237.126:40599 was elected leader, sending a full tablet report...
I20260129 03:00:53.299978 26292 ts_tablet_manager.cc:1434] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260129 03:00:53.300336 26292 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260129 03:00:53.300508 26294 tablet_bootstrap.cc:492] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: No bootstrap required, opened a new log
I20260129 03:00:53.300987 26294 ts_tablet_manager.cc:1403] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.010s
I20260129 03:00:53.301716 26202 heartbeater.cc:499] Master 127.22.237.126:40599 was elected leader, sending a full tablet report...
I20260129 03:00:53.302515 26293 ts_tablet_manager.cc:1434] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20260129 03:00:53.302874 26293 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260129 03:00:53.305394 26294 raft_consensus.cc:359] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.306133 26294 raft_consensus.cc:385] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:53.306434 26294 raft_consensus.cc:740] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa3e720de6004bd2893eabcd417658c2, State: Initialized, Role: FOLLOWER
I20260129 03:00:53.307297 26294 consensus_queue.cc:260] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.309450 26127 heartbeater.cc:499] Master 127.22.237.126:40599 was elected leader, sending a full tablet report...
I20260129 03:00:53.309979 26294 ts_tablet_manager.cc:1434] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
W20260129 03:00:53.310310 26294 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
W20260129 03:00:53.311108 26203 tablet.cc:2378] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260129 03:00:53.377871 26298 raft_consensus.cc:493] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:00:53.378315 26298 raft_consensus.cc:515] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.380561 26298 leader_election.cc:290] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e56b1cc76b147119bf29a3daff53e8b (127.22.237.66:43137), aa3e720de6004bd2893eabcd417658c2 (127.22.237.65:32977)
I20260129 03:00:53.389873 26177 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a94827a61f1748dca7119466aa9f0b14" candidate_uuid: "4c6e67967bb3473881a1721c9c923231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e56b1cc76b147119bf29a3daff53e8b" is_pre_election: true
I20260129 03:00:53.390509 26102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a94827a61f1748dca7119466aa9f0b14" candidate_uuid: "4c6e67967bb3473881a1721c9c923231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa3e720de6004bd2893eabcd417658c2" is_pre_election: true
I20260129 03:00:53.390551 26177 raft_consensus.cc:2468] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6e67967bb3473881a1721c9c923231 in term 0.
I20260129 03:00:53.391197 26102 raft_consensus.cc:2468] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c6e67967bb3473881a1721c9c923231 in term 0.
I20260129 03:00:53.391945 26216 leader_election.cc:304] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e56b1cc76b147119bf29a3daff53e8b, 4c6e67967bb3473881a1721c9c923231; no voters:
I20260129 03:00:53.392642 26298 raft_consensus.cc:2804] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 03:00:53.393004 26298 raft_consensus.cc:493] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:00:53.393230 26298 raft_consensus.cc:3060] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:53.398044 26298 raft_consensus.cc:515] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.399680 26298 leader_election.cc:290] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [CANDIDATE]: Term 1 election: Requested vote from peers 1e56b1cc76b147119bf29a3daff53e8b (127.22.237.66:43137), aa3e720de6004bd2893eabcd417658c2 (127.22.237.65:32977)
I20260129 03:00:53.400466 26177 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a94827a61f1748dca7119466aa9f0b14" candidate_uuid: "4c6e67967bb3473881a1721c9c923231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e56b1cc76b147119bf29a3daff53e8b"
I20260129 03:00:53.400732 26102 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a94827a61f1748dca7119466aa9f0b14" candidate_uuid: "4c6e67967bb3473881a1721c9c923231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa3e720de6004bd2893eabcd417658c2"
I20260129 03:00:53.401085 26177 raft_consensus.cc:3060] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:53.401248 26102 raft_consensus.cc:3060] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:53.408350 26102 raft_consensus.cc:2468] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6e67967bb3473881a1721c9c923231 in term 1.
I20260129 03:00:53.408350 26177 raft_consensus.cc:2468] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c6e67967bb3473881a1721c9c923231 in term 1.
I20260129 03:00:53.409955 26216 leader_election.cc:304] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e56b1cc76b147119bf29a3daff53e8b, 4c6e67967bb3473881a1721c9c923231; no voters:
I20260129 03:00:53.410744 26298 raft_consensus.cc:2804] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:53.411688 26298 raft_consensus.cc:697] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 1 LEADER]: Becoming Leader. State: Replica: 4c6e67967bb3473881a1721c9c923231, State: Running, Role: LEADER
I20260129 03:00:53.412457 26298 consensus_queue.cc:237] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.420408 25999 catalog_manager.cc:5654] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c6e67967bb3473881a1721c9c923231 (127.22.237.67). New cstate: current_term: 1 leader_uuid: "4c6e67967bb3473881a1721c9c923231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } health_report { overall_health: UNKNOWN } } }
I20260129 03:00:53.490505 26092 tablet_service.cc:1552] Processing DeleteTablet for tablet a94827a61f1748dca7119466aa9f0b14 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:35874
I20260129 03:00:53.492199 26305 tablet_replica.cc:333] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: stopping tablet replica
I20260129 03:00:53.492774 26305 raft_consensus.cc:2243] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:53.493217 26305 raft_consensus.cc:2272] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:53.495357 26305 ts_tablet_manager.cc:1916] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 03:00:53.505824 26305 ts_tablet_manager.cc:1929] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20260129 03:00:53.506183 26305 log.cc:1199] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Deleting WAL directory at /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-0-root/wals/a94827a61f1748dca7119466aa9f0b14
I20260129 03:00:53.509965 26306 ts_tablet_manager.cc:933] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Initiating tablet copy from peer 4c6e67967bb3473881a1721c9c923231 (127.22.237.67:46355)
I20260129 03:00:53.511528 26306 tablet_copy_client.cc:323] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: tablet copy: Beginning tablet copy session from remote peer at address 127.22.237.67:46355
I20260129 03:00:53.520689 26261 tablet_copy_service.cc:140] P 4c6e67967bb3473881a1721c9c923231: Received BeginTabletCopySession request for tablet a94827a61f1748dca7119466aa9f0b14 from peer aa3e720de6004bd2893eabcd417658c2 ({username='slave'} at 127.0.0.1:42058)
I20260129 03:00:53.521320 26261 tablet_copy_service.cc:161] P 4c6e67967bb3473881a1721c9c923231: Beginning new tablet copy session on tablet a94827a61f1748dca7119466aa9f0b14 from peer aa3e720de6004bd2893eabcd417658c2 at {username='slave'} at 127.0.0.1:42058: session id = aa3e720de6004bd2893eabcd417658c2-a94827a61f1748dca7119466aa9f0b14
I20260129 03:00:53.528829 26261 tablet_copy_source_session.cc:215] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 03:00:53.531389 26306 ts_tablet_manager.cc:1916] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260129 03:00:53.541616 26306 ts_tablet_manager.cc:1929] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20260129 03:00:53.542277 26306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94827a61f1748dca7119466aa9f0b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:53.547456 26306 tablet_copy_client.cc:806] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: tablet copy: Starting download of 0 data blocks...
I20260129 03:00:53.547986 26306 tablet_copy_client.cc:670] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: tablet copy: Starting download of 1 WAL segments...
I20260129 03:00:53.549616 25999 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34050:
table { table_id: "22037cf5fedf47b99911bca3a81110d6" table_name: "test-table" }
I20260129 03:00:53.550239 25999 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34050:
table { table_id: "22037cf5fedf47b99911bca3a81110d6" table_name: "test-table" }
I20260129 03:00:53.552047 26306 tablet_copy_client.cc:538] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 03:00:53.558308 26306 tablet_bootstrap.cc:492] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Bootstrap starting.
I20260129 03:00:53.559526 25999 catalog_manager.cc:5941] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45: Sending DeleteTablet for 3 replicas of tablet a94827a61f1748dca7119466aa9f0b14
I20260129 03:00:53.561386 26167 tablet_service.cc:1552] Processing DeleteTablet for tablet a94827a61f1748dca7119466aa9f0b14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-29 03:00:53 UTC) from {username='slave'} at 127.0.0.1:58476
I20260129 03:00:53.561551 26241 tablet_service.cc:1552] Processing DeleteTablet for tablet a94827a61f1748dca7119466aa9f0b14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-29 03:00:53 UTC) from {username='slave'} at 127.0.0.1:42032
I20260129 03:00:53.563172 26092 tablet_service.cc:1552] Processing DeleteTablet for tablet a94827a61f1748dca7119466aa9f0b14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-29 03:00:53 UTC) from {username='slave'} at 127.0.0.1:35858
I20260129 03:00:53.564883 26309 tablet_replica.cc:333] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: stopping tablet replica
I20260129 03:00:53.565133 26310 tablet_replica.cc:333] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: stopping tablet replica
W20260129 03:00:53.565340 25984 catalog_manager.cc:4965] TS aa3e720de6004bd2893eabcd417658c2 (127.22.237.65:32977): delete failed for tablet a94827a61f1748dca7119466aa9f0b14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a94827a61f1748dca7119466aa9f0b14 already in progress: copying tablet
I20260129 03:00:53.565732 26309 raft_consensus.cc:2243] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:53.565850 26310 raft_consensus.cc:2243] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:53.566390 26309 raft_consensus.cc:2272] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:53.566861 26310 pending_rounds.cc:70] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: Trying to abort 1 pending ops.
I20260129 03:00:53.567795 26310 pending_rounds.cc:77] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7248509556384165888 op_type: NO_OP noop_request { }
I20260129 03:00:53.568290 26310 raft_consensus.cc:2889] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260129 03:00:53.568641 26310 raft_consensus.cc:2272] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:53.569056 26309 ts_tablet_manager.cc:1916] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260129 03:00:53.571765 26310 ts_tablet_manager.cc:1916] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260129 03:00:53.573624 26306 tablet_bootstrap.cc:492] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260129 03:00:53.574366 26306 tablet_bootstrap.cc:492] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Bootstrap complete.
I20260129 03:00:53.574903 26306 ts_tablet_manager.cc:1403] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.001s
I20260129 03:00:53.576920 26306 raft_consensus.cc:359] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.577544 26306 raft_consensus.cc:740] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa3e720de6004bd2893eabcd417658c2, State: Initialized, Role: FOLLOWER
I20260129 03:00:53.578205 26306 consensus_queue.cc:260] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e56b1cc76b147119bf29a3daff53e8b" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 43137 } } peers { permanent_uuid: "4c6e67967bb3473881a1721c9c923231" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 46355 } } peers { permanent_uuid: "aa3e720de6004bd2893eabcd417658c2" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 32977 } }
I20260129 03:00:53.580029 26306 ts_tablet_manager.cc:1434] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260129 03:00:53.580274 26306 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260129 03:00:53.584805 26092 tablet_service.cc:1552] Processing DeleteTablet for tablet a94827a61f1748dca7119466aa9f0b14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-29 03:00:53 UTC) from {username='slave'} at 127.0.0.1:35858
I20260129 03:00:53.584827 26309 ts_tablet_manager.cc:1929] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260129 03:00:53.585407 26309 log.cc:1199] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: Deleting WAL directory at /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-1-root/wals/a94827a61f1748dca7119466aa9f0b14
I20260129 03:00:53.586128 26309 ts_tablet_manager.cc:1950] T a94827a61f1748dca7119466aa9f0b14 P 1e56b1cc76b147119bf29a3daff53e8b: Deleting consensus metadata
W20260129 03:00:53.586287 25984 catalog_manager.cc:4965] TS aa3e720de6004bd2893eabcd417658c2 (127.22.237.65:32977): delete failed for tablet a94827a61f1748dca7119466aa9f0b14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a94827a61f1748dca7119466aa9f0b14 already in progress: copying tablet
I20260129 03:00:53.586902 26310 ts_tablet_manager.cc:1929] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260129 03:00:53.587277 26310 log.cc:1199] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: Deleting WAL directory at /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-2-root/wals/a94827a61f1748dca7119466aa9f0b14
I20260129 03:00:53.588055 26310 ts_tablet_manager.cc:1950] T a94827a61f1748dca7119466aa9f0b14 P 4c6e67967bb3473881a1721c9c923231: Deleting consensus metadata
I20260129 03:00:53.589313 25986 catalog_manager.cc:4985] TS 1e56b1cc76b147119bf29a3daff53e8b (127.22.237.66:43137): tablet a94827a61f1748dca7119466aa9f0b14 (table test-table [id=22037cf5fedf47b99911bca3a81110d6]) successfully deleted
I20260129 03:00:53.589394 25985 catalog_manager.cc:4985] TS 4c6e67967bb3473881a1721c9c923231 (127.22.237.67:46355): tablet a94827a61f1748dca7119466aa9f0b14 (table test-table [id=22037cf5fedf47b99911bca3a81110d6]) successfully deleted
I20260129 03:00:53.632212 26261 tablet_copy_service.cc:342] P 4c6e67967bb3473881a1721c9c923231: Request end of tablet copy session aa3e720de6004bd2893eabcd417658c2-a94827a61f1748dca7119466aa9f0b14 received from {username='slave'} at 127.0.0.1:42058
I20260129 03:00:53.632602 26261 tablet_copy_service.cc:434] P 4c6e67967bb3473881a1721c9c923231: ending tablet copy session aa3e720de6004bd2893eabcd417658c2-a94827a61f1748dca7119466aa9f0b14 on tablet a94827a61f1748dca7119466aa9f0b14 with peer aa3e720de6004bd2893eabcd417658c2
I20260129 03:00:53.635771 26092 tablet_service.cc:1552] Processing DeleteTablet for tablet a94827a61f1748dca7119466aa9f0b14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-01-29 03:00:53 UTC) from {username='slave'} at 127.0.0.1:35858
I20260129 03:00:53.636514 26305 tablet_replica.cc:333] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: stopping tablet replica
I20260129 03:00:53.637084 26305 raft_consensus.cc:2243] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 03:00:53.637423 26305 pending_rounds.cc:70] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Trying to abort 1 pending ops.
I20260129 03:00:53.637630 26305 pending_rounds.cc:77] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7248509556384165888 op_type: NO_OP noop_request { }
I20260129 03:00:53.637956 26305 raft_consensus.cc:2889] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260129 03:00:53.638233 26305 raft_consensus.cc:2272] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:53.640254 26305 ts_tablet_manager.cc:1916] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260129 03:00:53.650230 26305 ts_tablet_manager.cc:1929] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260129 03:00:53.650547 26305 log.cc:1199] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Deleting WAL directory at /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1769655599110862-23477-0/minicluster-data/ts-0-root/wals/a94827a61f1748dca7119466aa9f0b14
I20260129 03:00:53.651125 26305 ts_tablet_manager.cc:1950] T a94827a61f1748dca7119466aa9f0b14 P aa3e720de6004bd2893eabcd417658c2: Deleting consensus metadata
I20260129 03:00:53.653738 25984 catalog_manager.cc:4985] TS aa3e720de6004bd2893eabcd417658c2 (127.22.237.65:32977): tablet a94827a61f1748dca7119466aa9f0b14 (table test-table [id=22037cf5fedf47b99911bca3a81110d6]) successfully deleted
I20260129 03:00:53.695943 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:00:53.714973 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:53.731452 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:00:53.740777 23477 tablet_server.cc:178] TabletServer@127.22.237.66:0 shutting down...
I20260129 03:00:53.758939 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:53.775707 23477 tablet_server.cc:195] TabletServer@127.22.237.66:0 shutdown complete.
I20260129 03:00:53.784210 23477 tablet_server.cc:178] TabletServer@127.22.237.67:0 shutting down...
I20260129 03:00:53.801924 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:53.819180 23477 tablet_server.cc:195] TabletServer@127.22.237.67:0 shutdown complete.
I20260129 03:00:53.828879 23477 master.cc:561] Master@127.22.237.126:40599 shutting down...
I20260129 03:00:53.846060 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:53.846607 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:53.846946 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P e4f6c06a38fa486b90e5caab3c5bdb45: stopping tablet replica
I20260129 03:00:53.866575 23477 master.cc:583] Master@127.22.237.126:40599 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1222 ms)
[----------] 4 tests from TsTabletManagerITest (44165 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20260129 03:00:53.892199 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:43087
I20260129 03:00:53.893314 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:53.898535 26316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:53.899519 26317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:53.900763 26319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:53.901404 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:53.902439 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:53.902684 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:53.902805 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655653902794 us; error 0 us; skew 500 ppm
I20260129 03:00:53.903312 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:53.905903 23477 webserver.cc:492] Webserver started at http://127.22.237.126:39973/ using document root <none> and password file <none>
I20260129 03:00:53.906371 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:53.906533 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:53.906780 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:53.907846 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "3a4b7f1cc4584cecb00908b4ebb3dcbd"
format_stamp: "Formatted at 2026-01-29 03:00:53 on dist-test-slave-1zcf"
I20260129 03:00:53.912878 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260129 03:00:53.916400 26324 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:53.917349 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 03:00:53.917740 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "3a4b7f1cc4584cecb00908b4ebb3dcbd"
format_stamp: "Formatted at 2026-01-29 03:00:53 on dist-test-slave-1zcf"
I20260129 03:00:53.918063 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:53.944231 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:53.945279 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:53.979918 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:43087
I20260129 03:00:53.980018 26375 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:43087 every 8 connection(s)
I20260129 03:00:53.983592 26376 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:53.993937 26376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd: Bootstrap starting.
I20260129 03:00:53.998310 26376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:54.002562 26376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd: No bootstrap required, opened a new log
I20260129 03:00:54.004529 26376 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4b7f1cc4584cecb00908b4ebb3dcbd" member_type: VOTER }
I20260129 03:00:54.004902 26376 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:54.005149 26376 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4b7f1cc4584cecb00908b4ebb3dcbd, State: Initialized, Role: FOLLOWER
I20260129 03:00:54.005730 26376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4b7f1cc4584cecb00908b4ebb3dcbd" member_type: VOTER }
I20260129 03:00:54.006176 26376 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:54.006388 26376 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:54.006634 26376 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:54.011020 26376 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4b7f1cc4584cecb00908b4ebb3dcbd" member_type: VOTER }
I20260129 03:00:54.011543 26376 leader_election.cc:304] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a4b7f1cc4584cecb00908b4ebb3dcbd; no voters:
I20260129 03:00:54.012713 26376 leader_election.cc:290] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:54.013048 26379 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:54.014333 26379 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 1 LEADER]: Becoming Leader. State: Replica: 3a4b7f1cc4584cecb00908b4ebb3dcbd, State: Running, Role: LEADER
I20260129 03:00:54.015000 26379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4b7f1cc4584cecb00908b4ebb3dcbd" member_type: VOTER }
I20260129 03:00:54.015574 26376 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:54.019047 26381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a4b7f1cc4584cecb00908b4ebb3dcbd. Latest consensus state: current_term: 1 leader_uuid: "3a4b7f1cc4584cecb00908b4ebb3dcbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4b7f1cc4584cecb00908b4ebb3dcbd" member_type: VOTER } }
I20260129 03:00:54.019650 26381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:54.019881 26380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a4b7f1cc4584cecb00908b4ebb3dcbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a4b7f1cc4584cecb00908b4ebb3dcbd" member_type: VOTER } }
I20260129 03:00:54.020391 26380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:54.022939 26385 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:54.030108 26385 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:54.031482 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:54.039124 26385 catalog_manager.cc:1357] Generated new cluster ID: 8245dcd6c85c4b1899680ed7fc726545
I20260129 03:00:54.039443 26385 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:54.053000 26385 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:54.054302 26385 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:54.068984 26385 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd: Generated new TSK 0
I20260129 03:00:54.069587 26385 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:54.098332 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:54.104045 26397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:54.105204 26398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:54.106523 26400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:54.106658 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:54.107542 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:54.107740 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:54.107867 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655654107853 us; error 0 us; skew 500 ppm
I20260129 03:00:54.108319 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:54.110677 23477 webserver.cc:492] Webserver started at http://127.22.237.65:40141/ using document root <none> and password file <none>
I20260129 03:00:54.111166 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:54.111327 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:54.111541 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:54.112672 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "9fc2c46431a14403be50e55801c03cd6"
format_stamp: "Formatted at 2026-01-29 03:00:54 on dist-test-slave-1zcf"
I20260129 03:00:54.117149 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260129 03:00:54.120716 26405 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:54.121490 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:54.121825 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "9fc2c46431a14403be50e55801c03cd6"
format_stamp: "Formatted at 2026-01-29 03:00:54 on dist-test-slave-1zcf"
I20260129 03:00:54.122120 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:54.140130 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:54.141290 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:54.146306 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:54.146632 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:54.146905 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:54.147110 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:54.196162 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:41967
I20260129 03:00:54.196249 26475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:41967 every 8 connection(s)
I20260129 03:00:54.200995 26476 heartbeater.cc:344] Connected to a master server at 127.22.237.126:43087
I20260129 03:00:54.201340 26476 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:54.202072 26476 heartbeater.cc:507] Master 127.22.237.126:43087 requested a full tablet report, sending...
I20260129 03:00:54.203939 26341 ts_manager.cc:194] Registered new tserver with Master: 9fc2c46431a14403be50e55801c03cd6 (127.22.237.65:41967)
I20260129 03:00:54.204207 23477 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004813233s
I20260129 03:00:54.205605 26341 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51756
I20260129 03:00:54.236402 26441 tablet_service.cc:1505] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260129 03:00:54.237761 26441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:54.250001 26486 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: Bootstrap starting.
I20260129 03:00:54.254861 26486 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:54.258983 26486 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: No bootstrap required, opened a new log
I20260129 03:00:54.259440 26486 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260129 03:00:54.261448 26486 raft_consensus.cc:359] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } }
I20260129 03:00:54.261932 26486 raft_consensus.cc:385] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:54.262130 26486 raft_consensus.cc:740] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc2c46431a14403be50e55801c03cd6, State: Initialized, Role: FOLLOWER
I20260129 03:00:54.262619 26486 consensus_queue.cc:260] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } }
I20260129 03:00:54.263074 26486 raft_consensus.cc:399] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:54.263273 26486 raft_consensus.cc:493] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:54.263485 26486 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:54.268612 26486 raft_consensus.cc:515] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } }
I20260129 03:00:54.269212 26486 leader_election.cc:304] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fc2c46431a14403be50e55801c03cd6; no voters:
I20260129 03:00:54.270393 26486 leader_election.cc:290] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:54.270745 26488 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:54.272145 26488 raft_consensus.cc:697] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 1 LEADER]: Becoming Leader. State: Replica: 9fc2c46431a14403be50e55801c03cd6, State: Running, Role: LEADER
I20260129 03:00:54.272272 26476 heartbeater.cc:499] Master 127.22.237.126:43087 was elected leader, sending a full tablet report...
I20260129 03:00:54.272789 26486 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260129 03:00:54.272930 26488 consensus_queue.cc:237] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } }
I20260129 03:00:54.278062 26489 tablet_replica.cc:442] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fc2c46431a14403be50e55801c03cd6" committed_config { opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } } }
I20260129 03:00:54.278694 26489 tablet_replica.cc:445] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: This TxnStatusTablet replica's current role is: LEADER
I20260129 03:00:54.278645 26490 tablet_replica.cc:442] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: TxnStatusTablet state changed. Reason: New leader 9fc2c46431a14403be50e55801c03cd6. Latest consensus state: current_term: 1 leader_uuid: "9fc2c46431a14403be50e55801c03cd6" committed_config { opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } } }
I20260129 03:00:54.279170 26490 tablet_replica.cc:445] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: This TxnStatusTablet replica's current role is: LEADER
W20260129 03:00:54.279359 26341 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:00:54.280407 26493 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260129 03:00:54.280927 26493 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260129 03:00:54.284165 26341 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:00:54.317821 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:00:54.337100 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:54.337918 23477 tablet_replica.cc:333] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: stopping tablet replica
I20260129 03:00:54.338465 23477 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:54.338943 23477 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:54.360379 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:00:54.371718 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:54.377494 26496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:54.379338 26497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:54.380914 26499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:54.381783 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:54.382966 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:54.383237 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:54.383437 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655654383420 us; error 0 us; skew 500 ppm
I20260129 03:00:54.384178 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:54.387022 23477 webserver.cc:492] Webserver started at http://127.22.237.65:40141/ using document root <none> and password file <none>
I20260129 03:00:54.387656 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:54.387882 23477 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260129 03:00:54.392676 23477 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 03:00:54.396303 26504 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:54.397171 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 03:00:54.397457 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "9fc2c46431a14403be50e55801c03cd6"
format_stamp: "Formatted at 2026-01-29 03:00:54 on dist-test-slave-1zcf"
I20260129 03:00:54.397794 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:54.414630 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:54.415817 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:54.423305 26513 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20260129 03:00:54.430383 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260129 03:00:54.430650 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260129 03:00:54.430934 23477 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260129 03:00:54.436995 23477 ts_tablet_manager.cc:616] Registered 1 tablets
I20260129 03:00:54.437283 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s user 0.003s sys 0.000s
I20260129 03:00:54.442029 26513 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: Bootstrap starting.
I20260129 03:00:54.480083 26513 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 03:00:54.481179 26513 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: Bootstrap complete.
I20260129 03:00:54.482007 26513 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: Time spent bootstrapping tablet: real 0.040s user 0.034s sys 0.004s
I20260129 03:00:54.484961 26513 raft_consensus.cc:359] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } }
I20260129 03:00:54.485596 26513 raft_consensus.cc:740] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc2c46431a14403be50e55801c03cd6, State: Initialized, Role: FOLLOWER
I20260129 03:00:54.486546 26513 consensus_queue.cc:260] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } }
I20260129 03:00:54.487248 26513 raft_consensus.cc:399] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:54.487571 26513 raft_consensus.cc:493] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:54.487916 26513 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:54.495617 26513 raft_consensus.cc:515] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } }
I20260129 03:00:54.496490 26513 leader_election.cc:304] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fc2c46431a14403be50e55801c03cd6; no voters:
I20260129 03:00:54.504783 26553 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 03:00:54.505807 26513 leader_election.cc:290] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260129 03:00:54.508970 26513 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6: Time spent starting tablet: real 0.027s user 0.024s sys 0.000s
I20260129 03:00:54.511999 26553 raft_consensus.cc:697] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 2 LEADER]: Becoming Leader. State: Replica: 9fc2c46431a14403be50e55801c03cd6, State: Running, Role: LEADER
I20260129 03:00:54.512938 26553 consensus_queue.cc:237] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } }
I20260129 03:00:54.516258 26563 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9fc2c46431a14403be50e55801c03cd6" committed_config { opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } } }
I20260129 03:00:54.517138 26563 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260129 03:00:54.516217 26561 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 9fc2c46431a14403be50e55801c03cd6. Latest consensus state: current_term: 2 leader_uuid: "9fc2c46431a14403be50e55801c03cd6" committed_config { opid_index: -1 peers { permanent_uuid: "9fc2c46431a14403be50e55801c03cd6" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 41967 } } }
I20260129 03:00:54.518251 26561 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260129 03:00:54.524298 26568 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260129 03:00:54.524907 26568 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260129 03:00:54.535084 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:41967
I20260129 03:00:54.535161 26581 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:41967 every 8 connection(s)
I20260129 03:00:54.540961 26582 heartbeater.cc:344] Connected to a master server at 127.22.237.126:43087
I20260129 03:00:54.541514 26582 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:54.542632 23477 tablet_server.cc:178] TabletServer@127.22.237.65:41967 shutting down...
I20260129 03:00:54.542626 26582 heartbeater.cc:507] Master 127.22.237.126:43087 requested a full tablet report, sending...
I20260129 03:00:54.547236 26341 ts_manager.cc:194] Re-registered known tserver with Master: 9fc2c46431a14403be50e55801c03cd6 (127.22.237.65:41967)
W20260129 03:00:54.547842 26341 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:00:54.549715 26341 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51792
I20260129 03:00:54.552316 26582 heartbeater.cc:499] Master 127.22.237.126:43087 was elected leader, sending a full tablet report...
W20260129 03:00:54.556725 26341 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:00:54.570989 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:54.572160 23477 tablet_replica.cc:333] stopping tablet replica
I20260129 03:00:54.572837 23477 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 2 LEADER]: Raft consensus shutting down.
I20260129 03:00:54.573437 23477 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 9fc2c46431a14403be50e55801c03cd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:54.585443 23477 tablet_server.cc:195] TabletServer@127.22.237.65:41967 shutdown complete.
I20260129 03:00:54.596009 23477 master.cc:561] Master@127.22.237.126:43087 shutting down...
I20260129 03:00:54.611548 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:54.612231 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:54.612615 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3a4b7f1cc4584cecb00908b4ebb3dcbd: stopping tablet replica
I20260129 03:00:54.632313 23477 master.cc:583] Master@127.22.237.126:43087 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (764 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20260129 03:00:54.656213 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:36861
I20260129 03:00:54.657588 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:54.662781 26588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:54.666042 26589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:54.666790 26591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:54.667135 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:54.667920 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:54.668113 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:54.668251 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655654668234 us; error 0 us; skew 500 ppm
I20260129 03:00:54.668752 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:54.671205 23477 webserver.cc:492] Webserver started at http://127.22.237.126:46445/ using document root <none> and password file <none>
I20260129 03:00:54.671670 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:54.671856 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:54.672096 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:54.673153 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "96c5b6ff335544f6a47f913cad1571a5"
format_stamp: "Formatted at 2026-01-29 03:00:54 on dist-test-slave-1zcf"
I20260129 03:00:54.677814 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260129 03:00:54.681351 26596 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:54.682277 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:54.682579 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "96c5b6ff335544f6a47f913cad1571a5"
format_stamp: "Formatted at 2026-01-29 03:00:54 on dist-test-slave-1zcf"
I20260129 03:00:54.682896 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:54.716966 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:54.718053 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:54.764664 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:36861
I20260129 03:00:54.764727 26647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:36861 every 8 connection(s)
I20260129 03:00:54.769098 26648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:54.784955 26648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5: Bootstrap starting.
I20260129 03:00:54.791072 26648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:54.795994 26648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5: No bootstrap required, opened a new log
I20260129 03:00:54.798663 26648 raft_consensus.cc:359] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5b6ff335544f6a47f913cad1571a5" member_type: VOTER }
I20260129 03:00:54.799190 26648 raft_consensus.cc:385] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:54.799485 26648 raft_consensus.cc:740] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96c5b6ff335544f6a47f913cad1571a5, State: Initialized, Role: FOLLOWER
I20260129 03:00:54.800155 26648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5b6ff335544f6a47f913cad1571a5" member_type: VOTER }
I20260129 03:00:54.800760 26648 raft_consensus.cc:399] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:54.801056 26648 raft_consensus.cc:493] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:54.801405 26648 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:54.808667 26648 raft_consensus.cc:515] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5b6ff335544f6a47f913cad1571a5" member_type: VOTER }
I20260129 03:00:54.809345 26648 leader_election.cc:304] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96c5b6ff335544f6a47f913cad1571a5; no voters:
I20260129 03:00:54.810722 26648 leader_election.cc:290] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:54.810994 26651 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:54.813009 26651 raft_consensus.cc:697] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 1 LEADER]: Becoming Leader. State: Replica: 96c5b6ff335544f6a47f913cad1571a5, State: Running, Role: LEADER
I20260129 03:00:54.813745 26651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5b6ff335544f6a47f913cad1571a5" member_type: VOTER }
I20260129 03:00:54.814162 26648 sys_catalog.cc:565] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:54.816395 26652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96c5b6ff335544f6a47f913cad1571a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5b6ff335544f6a47f913cad1571a5" member_type: VOTER } }
I20260129 03:00:54.816994 26652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:54.818064 26653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96c5b6ff335544f6a47f913cad1571a5. Latest consensus state: current_term: 1 leader_uuid: "96c5b6ff335544f6a47f913cad1571a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5b6ff335544f6a47f913cad1571a5" member_type: VOTER } }
I20260129 03:00:54.818554 26653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:54.829631 26656 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:54.835177 26656 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:54.838301 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:54.844774 26656 catalog_manager.cc:1357] Generated new cluster ID: 1f51f49dc87348adb0e8f327d6f049e9
I20260129 03:00:54.845063 26656 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:54.859880 26656 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:54.861608 26656 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:54.877982 26656 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5: Generated new TSK 0
I20260129 03:00:54.878746 26656 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:54.907585 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:54.915648 26669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:54.916785 26670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:54.919911 26672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:54.920039 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:54.921191 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:54.921406 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:54.921581 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655654921555 us; error 0 us; skew 500 ppm
I20260129 03:00:54.922138 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:54.924925 23477 webserver.cc:492] Webserver started at http://127.22.237.65:39123/ using document root <none> and password file <none>
I20260129 03:00:54.925408 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:54.925592 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:54.925904 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:54.927006 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "72b9df16ee864d8287df75879ca99572"
format_stamp: "Formatted at 2026-01-29 03:00:54 on dist-test-slave-1zcf"
I20260129 03:00:54.932020 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260129 03:00:54.935559 26677 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:54.936419 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260129 03:00:54.936805 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "72b9df16ee864d8287df75879ca99572"
format_stamp: "Formatted at 2026-01-29 03:00:54 on dist-test-slave-1zcf"
I20260129 03:00:54.937170 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:54.967882 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:54.969213 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:54.981590 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:54.981937 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260129 03:00:54.982236 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:54.982438 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:55.060437 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:39783
I20260129 03:00:55.060540 26747 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:39783 every 8 connection(s)
I20260129 03:00:55.065431 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260129 03:00:55.065626 26748 heartbeater.cc:344] Connected to a master server at 127.22.237.126:36861
I20260129 03:00:55.066094 26748 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:55.066972 26748 heartbeater.cc:507] Master 127.22.237.126:36861 requested a full tablet report, sending...
I20260129 03:00:55.069288 26613 ts_manager.cc:194] Registered new tserver with Master: 72b9df16ee864d8287df75879ca99572 (127.22.237.65:39783)
W20260129 03:00:55.071833 26750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:55.071836 26613 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33326
W20260129 03:00:55.074002 26751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:55.079851 26753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:55.080542 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:55.081315 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:55.081517 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:55.081703 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655655081682 us; error 0 us; skew 500 ppm
I20260129 03:00:55.082223 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:55.084818 23477 webserver.cc:492] Webserver started at http://127.22.237.66:37517/ using document root <none> and password file <none>
I20260129 03:00:55.085297 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:55.085480 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:55.085786 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:55.086864 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-1-root/instance:
uuid: "f845e5cea1224f66a1b965313480f629"
format_stamp: "Formatted at 2026-01-29 03:00:55 on dist-test-slave-1zcf"
I20260129 03:00:55.091673 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 03:00:55.095428 26758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:55.096331 23477 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260129 03:00:55.096663 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-1-root
uuid: "f845e5cea1224f66a1b965313480f629"
format_stamp: "Formatted at 2026-01-29 03:00:55 on dist-test-slave-1zcf"
I20260129 03:00:55.096937 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:55.111802 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:55.112927 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:55.118922 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:55.119235 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:55.119529 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:55.119684 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:55.189436 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.66:32995
I20260129 03:00:55.189533 26827 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.66:32995 every 8 connection(s)
I20260129 03:00:55.194422 26828 heartbeater.cc:344] Connected to a master server at 127.22.237.126:36861
I20260129 03:00:55.194794 26828 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:55.195576 26828 heartbeater.cc:507] Master 127.22.237.126:36861 requested a full tablet report, sending...
I20260129 03:00:55.197402 26613 ts_manager.cc:194] Registered new tserver with Master: f845e5cea1224f66a1b965313480f629 (127.22.237.66:32995)
I20260129 03:00:55.197885 23477 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.005064528s
I20260129 03:00:55.198920 26613 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33338
I20260129 03:00:55.233098 26713 tablet_service.cc:1505] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260129 03:00:55.234294 26713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:55.245723 26838 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: Bootstrap starting.
I20260129 03:00:55.250123 26838 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:55.254292 26838 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: No bootstrap required, opened a new log
I20260129 03:00:55.254678 26838 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260129 03:00:55.256578 26838 raft_consensus.cc:359] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } }
I20260129 03:00:55.257006 26838 raft_consensus.cc:385] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:55.257218 26838 raft_consensus.cc:740] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b9df16ee864d8287df75879ca99572, State: Initialized, Role: FOLLOWER
I20260129 03:00:55.257740 26838 consensus_queue.cc:260] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } }
I20260129 03:00:55.258203 26838 raft_consensus.cc:399] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:55.258423 26838 raft_consensus.cc:493] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:55.258671 26838 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:55.262986 26838 raft_consensus.cc:515] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } }
I20260129 03:00:55.263547 26838 leader_election.cc:304] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72b9df16ee864d8287df75879ca99572; no voters:
I20260129 03:00:55.264897 26838 leader_election.cc:290] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:55.265240 26840 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:55.266918 26748 heartbeater.cc:499] Master 127.22.237.126:36861 was elected leader, sending a full tablet report...
I20260129 03:00:55.267537 26840 raft_consensus.cc:697] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 1 LEADER]: Becoming Leader. State: Replica: 72b9df16ee864d8287df75879ca99572, State: Running, Role: LEADER
I20260129 03:00:55.268471 26838 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260129 03:00:55.268286 26840 consensus_queue.cc:237] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } }
I20260129 03:00:55.271377 26841 tablet_replica.cc:442] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72b9df16ee864d8287df75879ca99572" committed_config { opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } } }
I20260129 03:00:55.271934 26841 tablet_replica.cc:445] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: This TxnStatusTablet replica's current role is: LEADER
I20260129 03:00:55.273329 26842 tablet_replica.cc:442] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: TxnStatusTablet state changed. Reason: New leader 72b9df16ee864d8287df75879ca99572. Latest consensus state: current_term: 1 leader_uuid: "72b9df16ee864d8287df75879ca99572" committed_config { opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } } }
I20260129 03:00:55.273895 26842 tablet_replica.cc:445] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: This TxnStatusTablet replica's current role is: LEADER
W20260129 03:00:55.274381 26613 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:00:55.278764 26844 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260129 03:00:55.279196 26844 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260129 03:00:55.279301 26613 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:00:55.332605 26849 ts_tablet_manager.cc:933] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: Initiating tablet copy from peer 72b9df16ee864d8287df75879ca99572 (127.22.237.65:39783)
I20260129 03:00:55.334236 26849 tablet_copy_client.cc:323] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: tablet copy: Beginning tablet copy session from remote peer at address 127.22.237.65:39783
I20260129 03:00:55.342729 26733 tablet_copy_service.cc:140] P 72b9df16ee864d8287df75879ca99572: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer f845e5cea1224f66a1b965313480f629 ({username='slave'} at 127.0.0.1:51744)
I20260129 03:00:55.343228 26733 tablet_copy_service.cc:161] P 72b9df16ee864d8287df75879ca99572: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer f845e5cea1224f66a1b965313480f629 at {username='slave'} at 127.0.0.1:51744: session id = f845e5cea1224f66a1b965313480f629-11111111111111111111111111111111
I20260129 03:00:55.347910 26733 tablet_copy_source_session.cc:215] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 03:00:55.350513 26849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:55.360657 26849 tablet_copy_client.cc:806] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: tablet copy: Starting download of 0 data blocks...
I20260129 03:00:55.361196 26849 tablet_copy_client.cc:670] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: tablet copy: Starting download of 1 WAL segments...
I20260129 03:00:55.364473 26849 tablet_copy_client.cc:538] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 03:00:55.370363 26849 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: Bootstrap starting.
I20260129 03:00:55.392562 26849 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 03:00:55.393292 26849 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: Bootstrap complete.
I20260129 03:00:55.393916 26849 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.001s
I20260129 03:00:55.395748 26849 raft_consensus.cc:359] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } }
I20260129 03:00:55.396169 26849 raft_consensus.cc:740] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: f845e5cea1224f66a1b965313480f629, State: Initialized, Role: NON_PARTICIPANT
I20260129 03:00:55.396615 26849 consensus_queue.cc:260] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } }
I20260129 03:00:55.398321 26852 tablet_replica.cc:442] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } } }
I20260129 03:00:55.398531 26828 heartbeater.cc:499] Master 127.22.237.126:36861 was elected leader, sending a full tablet report...
I20260129 03:00:55.398932 26852 tablet_replica.cc:445] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20260129 03:00:55.399187 26849 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260129 03:00:55.400910 26733 tablet_copy_service.cc:342] P 72b9df16ee864d8287df75879ca99572: Request end of tablet copy session f845e5cea1224f66a1b965313480f629-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:51744
I20260129 03:00:55.401297 26733 tablet_copy_service.cc:434] P 72b9df16ee864d8287df75879ca99572: ending tablet copy session f845e5cea1224f66a1b965313480f629-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer f845e5cea1224f66a1b965313480f629
W20260129 03:00:55.401559 26613 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260129 03:00:55.405419 26613 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:00:55.425930 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:00:55.449874 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:55.450699 23477 tablet_replica.cc:333] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: stopping tablet replica
I20260129 03:00:55.451319 23477 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:55.451893 23477 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:55.463271 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:00:55.473716 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:55.479616 26853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:55.479656 26854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:55.482336 26856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:55.487438 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:55.488430 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:55.488651 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:55.488808 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655655488792 us; error 0 us; skew 500 ppm
I20260129 03:00:55.489311 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:55.492178 23477 webserver.cc:492] Webserver started at http://127.22.237.65:39123/ using document root <none> and password file <none>
I20260129 03:00:55.492646 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:55.492810 23477 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260129 03:00:55.496487 23477 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260129 03:00:55.499414 26861 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:55.500232 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260129 03:00:55.500599 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "72b9df16ee864d8287df75879ca99572"
format_stamp: "Formatted at 2026-01-29 03:00:54 on dist-test-slave-1zcf"
I20260129 03:00:55.500958 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:55.523430 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:55.524768 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:55.532490 26870 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20260129 03:00:55.541795 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260129 03:00:55.542133 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s user 0.001s sys 0.000s
I20260129 03:00:55.542438 23477 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260129 03:00:55.548126 23477 ts_tablet_manager.cc:616] Registered 1 tablets
I20260129 03:00:55.548373 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s user 0.004s sys 0.000s
I20260129 03:00:55.548473 26870 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: Bootstrap starting.
I20260129 03:00:55.585353 26870 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 03:00:55.586588 26870 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: Bootstrap complete.
I20260129 03:00:55.587481 26870 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.005s
I20260129 03:00:55.590265 26870 raft_consensus.cc:359] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } }
I20260129 03:00:55.590901 26870 raft_consensus.cc:740] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b9df16ee864d8287df75879ca99572, State: Initialized, Role: FOLLOWER
I20260129 03:00:55.591595 26870 consensus_queue.cc:260] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } }
I20260129 03:00:55.592859 26870 raft_consensus.cc:399] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:55.593209 26870 raft_consensus.cc:493] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:55.593616 26870 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 1 FOLLOWER]: Advancing to term 2
I20260129 03:00:55.604480 26870 raft_consensus.cc:515] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } }
I20260129 03:00:55.605319 26870 leader_election.cc:304] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72b9df16ee864d8287df75879ca99572; no voters:
I20260129 03:00:55.622587 26870 leader_election.cc:290] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260129 03:00:55.622849 26882 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 2 FOLLOWER]: Leader election won for term 2
I20260129 03:00:55.633170 26870 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572: Time spent starting tablet: real 0.045s user 0.021s sys 0.018s
I20260129 03:00:55.634768 26882 raft_consensus.cc:697] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 2 LEADER]: Becoming Leader. State: Replica: 72b9df16ee864d8287df75879ca99572, State: Running, Role: LEADER
I20260129 03:00:55.635591 26882 consensus_queue.cc:237] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } }
I20260129 03:00:55.639166 26885 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "72b9df16ee864d8287df75879ca99572" committed_config { opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } } }
I20260129 03:00:55.639861 26885 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260129 03:00:55.641512 26886 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 72b9df16ee864d8287df75879ca99572. Latest consensus state: current_term: 2 leader_uuid: "72b9df16ee864d8287df75879ca99572" committed_config { opid_index: -1 peers { permanent_uuid: "72b9df16ee864d8287df75879ca99572" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 39783 } } }
I20260129 03:00:55.642058 26886 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260129 03:00:55.646111 26893 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260129 03:00:55.646574 26893 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260129 03:00:55.684314 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:39783
I20260129 03:00:55.684410 26938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:39783 every 8 connection(s)
I20260129 03:00:55.690078 26939 heartbeater.cc:344] Connected to a master server at 127.22.237.126:36861
I20260129 03:00:55.690533 26939 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:55.691435 26939 heartbeater.cc:507] Master 127.22.237.126:36861 requested a full tablet report, sending...
I20260129 03:00:55.693753 23477 tablet_server.cc:178] TabletServer@127.22.237.65:39783 shutting down...
I20260129 03:00:55.694795 26613 ts_manager.cc:194] Re-registered known tserver with Master: 72b9df16ee864d8287df75879ca99572 (127.22.237.65:39783)
W20260129 03:00:55.695482 26613 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:00:55.697265 26613 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33376
I20260129 03:00:55.702149 26939 heartbeater.cc:499] Master 127.22.237.126:36861 was elected leader, sending a full tablet report...
W20260129 03:00:55.705207 26613 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:00:55.725525 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:55.726367 23477 tablet_replica.cc:333] stopping tablet replica
I20260129 03:00:55.726902 23477 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 2 LEADER]: Raft consensus shutting down.
I20260129 03:00:55.727316 23477 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 72b9df16ee864d8287df75879ca99572 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:55.737707 23477 tablet_server.cc:195] TabletServer@127.22.237.65:39783 shutdown complete.
I20260129 03:00:55.748519 23477 tablet_server.cc:178] TabletServer@127.22.237.66:0 shutting down...
I20260129 03:00:55.766273 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:55.767079 23477 tablet_replica.cc:333] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629: stopping tablet replica
I20260129 03:00:55.767540 23477 raft_consensus.cc:2243] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260129 03:00:55.767877 23477 raft_consensus.cc:2272] T 11111111111111111111111111111111 P f845e5cea1224f66a1b965313480f629 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260129 03:00:55.786979 23477 tablet_server.cc:195] TabletServer@127.22.237.66:0 shutdown complete.
I20260129 03:00:55.800149 23477 master.cc:561] Master@127.22.237.126:36861 shutting down...
I20260129 03:00:55.818579 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:55.819226 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:55.819583 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 96c5b6ff335544f6a47f913cad1571a5: stopping tablet replica
I20260129 03:00:55.839042 23477 master.cc:583] Master@127.22.237.126:36861 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1209 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20260129 03:00:55.866128 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:43081
I20260129 03:00:55.867172 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:55.872220 26945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:55.874080 26946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:55.875314 26948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:55.875855 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:55.876633 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:55.876801 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:55.876942 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655655876929 us; error 0 us; skew 500 ppm
I20260129 03:00:55.877473 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:55.879871 23477 webserver.cc:492] Webserver started at http://127.22.237.126:43609/ using document root <none> and password file <none>
I20260129 03:00:55.880326 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:55.880491 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:55.880726 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:55.881765 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "9195c5d7db6e44b18fdf8f137a2e5890"
format_stamp: "Formatted at 2026-01-29 03:00:55 on dist-test-slave-1zcf"
I20260129 03:00:55.886189 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260129 03:00:55.890002 26953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:55.890872 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260129 03:00:55.891161 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "9195c5d7db6e44b18fdf8f137a2e5890"
format_stamp: "Formatted at 2026-01-29 03:00:55 on dist-test-slave-1zcf"
I20260129 03:00:55.891426 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:55.912827 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:55.913841 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:55.956754 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:43081
I20260129 03:00:55.956854 27004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:43081 every 8 connection(s)
I20260129 03:00:55.961104 27005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:55.976085 27005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890: Bootstrap starting.
I20260129 03:00:55.982264 27005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:55.987118 27005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890: No bootstrap required, opened a new log
I20260129 03:00:55.989778 27005 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9195c5d7db6e44b18fdf8f137a2e5890" member_type: VOTER }
I20260129 03:00:55.990285 27005 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:55.990562 27005 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9195c5d7db6e44b18fdf8f137a2e5890, State: Initialized, Role: FOLLOWER
I20260129 03:00:55.991201 27005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9195c5d7db6e44b18fdf8f137a2e5890" member_type: VOTER }
I20260129 03:00:55.991791 27005 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:55.992084 27005 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:55.992399 27005 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:55.999110 27005 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9195c5d7db6e44b18fdf8f137a2e5890" member_type: VOTER }
I20260129 03:00:55.999784 27005 leader_election.cc:304] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9195c5d7db6e44b18fdf8f137a2e5890; no voters:
I20260129 03:00:56.001096 27005 leader_election.cc:290] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:56.001368 27008 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:56.004446 27005 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:56.005236 27008 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 1 LEADER]: Becoming Leader. State: Replica: 9195c5d7db6e44b18fdf8f137a2e5890, State: Running, Role: LEADER
I20260129 03:00:56.005959 27008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9195c5d7db6e44b18fdf8f137a2e5890" member_type: VOTER }
I20260129 03:00:56.008285 27009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9195c5d7db6e44b18fdf8f137a2e5890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9195c5d7db6e44b18fdf8f137a2e5890" member_type: VOTER } }
I20260129 03:00:56.008679 27009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:56.009352 27010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9195c5d7db6e44b18fdf8f137a2e5890. Latest consensus state: current_term: 1 leader_uuid: "9195c5d7db6e44b18fdf8f137a2e5890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9195c5d7db6e44b18fdf8f137a2e5890" member_type: VOTER } }
I20260129 03:00:56.010006 27010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:56.025326 27013 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:56.030637 27013 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:56.034541 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:56.040742 27013 catalog_manager.cc:1357] Generated new cluster ID: 04bbba2dbbb04220ab53bff3cee6eb23
I20260129 03:00:56.041002 27013 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:56.060400 27013 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:56.061599 27013 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:56.079550 27013 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890: Generated new TSK 0
I20260129 03:00:56.080133 27013 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:56.101945 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:56.108893 27026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:56.110386 27027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:56.114970 23477 server_base.cc:1060] running on GCE node
W20260129 03:00:56.116641 27029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:56.117748 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:56.118023 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:56.118219 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655656118202 us; error 0 us; skew 500 ppm
I20260129 03:00:56.118942 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:56.121740 23477 webserver.cc:492] Webserver started at http://127.22.237.65:46227/ using document root <none> and password file <none>
I20260129 03:00:56.122371 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:56.122613 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:56.122937 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:56.124359 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "d063ac5743ef4f5490e956630600256a"
format_stamp: "Formatted at 2026-01-29 03:00:56 on dist-test-slave-1zcf"
I20260129 03:00:56.130435 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260129 03:00:56.134712 27034 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:56.135617 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260129 03:00:56.135902 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "d063ac5743ef4f5490e956630600256a"
format_stamp: "Formatted at 2026-01-29 03:00:56 on dist-test-slave-1zcf"
I20260129 03:00:56.136180 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:56.159513 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:56.160540 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:56.180285 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:56.180531 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:56.180801 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:56.180969 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:56.332526 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:43453
I20260129 03:00:56.332643 27104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:43453 every 8 connection(s)
I20260129 03:00:56.365341 27105 heartbeater.cc:344] Connected to a master server at 127.22.237.126:43081
I20260129 03:00:56.365828 27105 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:56.366688 27105 heartbeater.cc:507] Master 127.22.237.126:43081 requested a full tablet report, sending...
I20260129 03:00:56.368681 26970 ts_manager.cc:194] Registered new tserver with Master: d063ac5743ef4f5490e956630600256a (127.22.237.65:43453)
I20260129 03:00:56.369257 23477 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.031292114s
I20260129 03:00:56.371035 26970 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46402
I20260129 03:00:56.405310 27070 tablet_service.cc:1505] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260129 03:00:56.406869 27070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:56.423346 27115 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a: Bootstrap starting.
I20260129 03:00:56.427701 27115 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:56.431768 27115 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a: No bootstrap required, opened a new log
I20260129 03:00:56.432152 27115 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260129 03:00:56.434012 27115 raft_consensus.cc:359] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "d063ac5743ef4f5490e956630600256a" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43453 } }
I20260129 03:00:56.434391 27115 raft_consensus.cc:385] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:56.434604 27115 raft_consensus.cc:740] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d063ac5743ef4f5490e956630600256a, State: Initialized, Role: FOLLOWER
I20260129 03:00:56.435093 27115 consensus_queue.cc:260] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "d063ac5743ef4f5490e956630600256a" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43453 } }
I20260129 03:00:56.435542 27115 raft_consensus.cc:399] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:56.435755 27115 raft_consensus.cc:493] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:56.435989 27115 raft_consensus.cc:3060] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:56.440431 27115 raft_consensus.cc:515] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "d063ac5743ef4f5490e956630600256a" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43453 } }
I20260129 03:00:56.440927 27115 leader_election.cc:304] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d063ac5743ef4f5490e956630600256a; no voters:
I20260129 03:00:56.442219 27115 leader_election.cc:290] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:56.442533 27117 raft_consensus.cc:2804] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:56.444032 27105 heartbeater.cc:499] Master 127.22.237.126:43081 was elected leader, sending a full tablet report...
I20260129 03:00:56.444708 27115 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260129 03:00:56.445185 27117 raft_consensus.cc:697] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 1 LEADER]: Becoming Leader. State: Replica: d063ac5743ef4f5490e956630600256a, State: Running, Role: LEADER
I20260129 03:00:56.445871 27117 consensus_queue.cc:237] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "d063ac5743ef4f5490e956630600256a" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43453 } }
I20260129 03:00:56.453289 27118 tablet_replica.cc:442] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d063ac5743ef4f5490e956630600256a" committed_config { opid_index: -1 peers { permanent_uuid: "d063ac5743ef4f5490e956630600256a" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43453 } } }
I20260129 03:00:56.453492 27119 tablet_replica.cc:442] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a: TxnStatusTablet state changed. Reason: New leader d063ac5743ef4f5490e956630600256a. Latest consensus state: current_term: 1 leader_uuid: "d063ac5743ef4f5490e956630600256a" committed_config { opid_index: -1 peers { permanent_uuid: "d063ac5743ef4f5490e956630600256a" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43453 } } }
I20260129 03:00:56.453907 27118 tablet_replica.cc:445] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a: This TxnStatusTablet replica's current role is: LEADER
W20260129 03:00:56.454108 26970 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:00:56.454169 27119 tablet_replica.cc:445] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a: This TxnStatusTablet replica's current role is: LEADER
I20260129 03:00:56.455577 27122 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260129 03:00:56.456125 27122 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260129 03:00:56.458573 26970 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260129 03:00:56.511574 27057 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260129 03:00:56.512020 27057 txn_status_manager.cc:244] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260129 03:00:56.512423 27057 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20260129 03:00:56.520431 27057 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260129 03:00:56.562223 27057 txn_status_manager.cc:244] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260129 03:00:56.562508 27057 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20260129 03:00:56.572927 27057 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20260129 03:00:56.588136 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:47774
W20260129 03:00:56.590914 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:47774
W20260129 03:00:56.593199 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:47774
W20260129 03:00:56.596036 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:47774
W20260129 03:00:56.598336 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:47774
W20260129 03:00:56.608078 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:47774
W20260129 03:00:56.622174 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:47790
W20260129 03:00:56.624225 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:47790
W20260129 03:00:56.626230 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:47790
W20260129 03:00:56.628120 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:47790
W20260129 03:00:56.630000 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:47790
W20260129 03:00:56.631875 27070 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:47790
I20260129 03:00:56.635035 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:00:56.661244 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:00:56.661988 23477 tablet_replica.cc:333] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a: stopping tablet replica
I20260129 03:00:56.662499 23477 raft_consensus.cc:2243] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:56.662953 23477 raft_consensus.cc:2272] T 11111111111111111111111111111111 P d063ac5743ef4f5490e956630600256a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:56.673557 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:00:56.683702 23477 master.cc:561] Master@127.22.237.126:43081 shutting down...
I20260129 03:00:56.698285 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:00:56.698786 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:00:56.699061 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9195c5d7db6e44b18fdf8f137a2e5890: stopping tablet replica
W20260129 03:00:57.285327 27001 debug-util.cc:398] Leaking SignalData structure 0x7b08000a51e0 after lost signal to thread 23480
W20260129 03:00:57.714614 23477 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 27001)
W20260129 03:00:58.715065 23477 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 27001)
I20260129 03:00:59.633026 23477 master.cc:583] Master@127.22.237.126:43081 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (3790 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20260129 03:00:59.656030 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:36137
I20260129 03:00:59.657071 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:59.661854 27130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:59.662796 27131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:59.664095 27133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:59.664433 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:59.667665 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:59.667843 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:59.667953 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655659667942 us; error 0 us; skew 500 ppm
I20260129 03:00:59.668406 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:59.670717 23477 webserver.cc:492] Webserver started at http://127.22.237.126:41445/ using document root <none> and password file <none>
I20260129 03:00:59.671144 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:59.671290 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:59.671494 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:59.672474 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "9242f521b30e47aa841c4e373b4d7965"
format_stamp: "Formatted at 2026-01-29 03:00:59 on dist-test-slave-1zcf"
I20260129 03:00:59.676808 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260129 03:00:59.680040 27138 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:59.680944 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 03:00:59.681277 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "9242f521b30e47aa841c4e373b4d7965"
format_stamp: "Formatted at 2026-01-29 03:00:59 on dist-test-slave-1zcf"
I20260129 03:00:59.681531 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:59.699726 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:59.700699 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:59.735353 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:36137
I20260129 03:00:59.735448 27189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:36137 every 8 connection(s)
I20260129 03:00:59.738961 27190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:00:59.748813 27190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965: Bootstrap starting.
I20260129 03:00:59.753067 27190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965: Neither blocks nor log segments found. Creating new log.
I20260129 03:00:59.756834 27190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965: No bootstrap required, opened a new log
I20260129 03:00:59.758807 27190 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9242f521b30e47aa841c4e373b4d7965" member_type: VOTER }
I20260129 03:00:59.759191 27190 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:00:59.759414 27190 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9242f521b30e47aa841c4e373b4d7965, State: Initialized, Role: FOLLOWER
I20260129 03:00:59.759951 27190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9242f521b30e47aa841c4e373b4d7965" member_type: VOTER }
I20260129 03:00:59.760377 27190 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:00:59.760581 27190 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:00:59.760828 27190 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:00:59.765566 27190 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9242f521b30e47aa841c4e373b4d7965" member_type: VOTER }
I20260129 03:00:59.766132 27190 leader_election.cc:304] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9242f521b30e47aa841c4e373b4d7965; no voters:
I20260129 03:00:59.767179 27190 leader_election.cc:290] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:00:59.767521 27193 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:00:59.768767 27193 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 1 LEADER]: Becoming Leader. State: Replica: 9242f521b30e47aa841c4e373b4d7965, State: Running, Role: LEADER
I20260129 03:00:59.769421 27193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9242f521b30e47aa841c4e373b4d7965" member_type: VOTER }
I20260129 03:00:59.770066 27190 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:00:59.773874 27195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9242f521b30e47aa841c4e373b4d7965. Latest consensus state: current_term: 1 leader_uuid: "9242f521b30e47aa841c4e373b4d7965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9242f521b30e47aa841c4e373b4d7965" member_type: VOTER } }
I20260129 03:00:59.774931 27195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:59.776973 27194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9242f521b30e47aa841c4e373b4d7965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9242f521b30e47aa841c4e373b4d7965" member_type: VOTER } }
I20260129 03:00:59.777648 27194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [sys.catalog]: This master's current role is: LEADER
I20260129 03:00:59.779439 27199 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:00:59.784559 27199 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:00:59.786227 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:00:59.792824 27199 catalog_manager.cc:1357] Generated new cluster ID: 79bd4811fa9b4753805a538e54b55531
I20260129 03:00:59.793136 27199 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:00:59.812372 27199 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:00:59.813645 27199 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:00:59.837316 27199 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965: Generated new TSK 0
I20260129 03:00:59.837972 27199 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:00:59.852440 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:00:59.858068 27211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:59.858888 27212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:00:59.860855 27214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:00:59.861208 23477 server_base.cc:1060] running on GCE node
I20260129 03:00:59.862053 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:00:59.862236 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:00:59.862383 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655659862370 us; error 0 us; skew 500 ppm
I20260129 03:00:59.862912 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:00:59.865192 23477 webserver.cc:492] Webserver started at http://127.22.237.65:42323/ using document root <none> and password file <none>
I20260129 03:00:59.865688 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:00:59.865861 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:00:59.866111 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:00:59.867153 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "09a7f630b7864ad4a148402b48d1702c"
format_stamp: "Formatted at 2026-01-29 03:00:59 on dist-test-slave-1zcf"
I20260129 03:00:59.871572 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 03:00:59.875077 27219 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:59.875845 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:00:59.876152 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "09a7f630b7864ad4a148402b48d1702c"
format_stamp: "Formatted at 2026-01-29 03:00:59 on dist-test-slave-1zcf"
I20260129 03:00:59.876410 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:00:59.904127 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:00:59.905192 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:00:59.909945 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:00:59.910362 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:59.910732 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:00:59.910989 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:00:59.957861 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:42195
I20260129 03:00:59.957963 27289 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:42195 every 8 connection(s)
I20260129 03:00:59.962168 27290 heartbeater.cc:344] Connected to a master server at 127.22.237.126:36137
I20260129 03:00:59.962515 27290 heartbeater.cc:461] Registering TS with master...
I20260129 03:00:59.963217 27290 heartbeater.cc:507] Master 127.22.237.126:36137 requested a full tablet report, sending...
I20260129 03:00:59.965071 27155 ts_manager.cc:194] Registered new tserver with Master: 09a7f630b7864ad4a148402b48d1702c (127.22.237.65:42195)
I20260129 03:00:59.965651 23477 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00482477s
I20260129 03:00:59.966784 27155 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42186
I20260129 03:00:59.996078 27255 tablet_service.cc:1505] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260129 03:00:59.997170 27255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:00.008291 27300 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c: Bootstrap starting.
I20260129 03:01:00.012689 27300 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c: Neither blocks nor log segments found. Creating new log.
I20260129 03:01:00.018033 27300 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c: No bootstrap required, opened a new log
I20260129 03:01:00.018513 27300 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260129 03:01:00.042435 27300 raft_consensus.cc:359] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "09a7f630b7864ad4a148402b48d1702c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 42195 } }
I20260129 03:01:00.043877 27300 raft_consensus.cc:385] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:01:00.044198 27300 raft_consensus.cc:740] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7f630b7864ad4a148402b48d1702c, State: Initialized, Role: FOLLOWER
I20260129 03:01:00.044927 27300 consensus_queue.cc:260] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "09a7f630b7864ad4a148402b48d1702c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 42195 } }
I20260129 03:01:00.045389 27300 raft_consensus.cc:399] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:01:00.045584 27300 raft_consensus.cc:493] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:01:00.045873 27300 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:01:00.053166 27300 raft_consensus.cc:515] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "09a7f630b7864ad4a148402b48d1702c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 42195 } }
I20260129 03:01:00.053786 27300 leader_election.cc:304] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09a7f630b7864ad4a148402b48d1702c; no voters:
I20260129 03:01:00.055086 27300 leader_election.cc:290] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:01:00.055532 27303 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:01:00.057006 27303 raft_consensus.cc:697] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 1 LEADER]: Becoming Leader. State: Replica: 09a7f630b7864ad4a148402b48d1702c, State: Running, Role: LEADER
I20260129 03:01:00.057453 27290 heartbeater.cc:499] Master 127.22.237.126:36137 was elected leader, sending a full tablet report...
I20260129 03:01:00.057837 27300 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c: Time spent starting tablet: real 0.039s user 0.014s sys 0.006s
I20260129 03:01:00.057984 27303 consensus_queue.cc:237] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "09a7f630b7864ad4a148402b48d1702c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 42195 } }
I20260129 03:01:00.061813 27305 tablet_replica.cc:442] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c: TxnStatusTablet state changed. Reason: New leader 09a7f630b7864ad4a148402b48d1702c. Latest consensus state: current_term: 1 leader_uuid: "09a7f630b7864ad4a148402b48d1702c" committed_config { opid_index: -1 peers { permanent_uuid: "09a7f630b7864ad4a148402b48d1702c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 42195 } } }
I20260129 03:01:00.062103 27304 tablet_replica.cc:442] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09a7f630b7864ad4a148402b48d1702c" committed_config { opid_index: -1 peers { permanent_uuid: "09a7f630b7864ad4a148402b48d1702c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 42195 } } }
I20260129 03:01:00.062490 27305 tablet_replica.cc:445] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c: This TxnStatusTablet replica's current role is: LEADER
I20260129 03:01:00.062700 27304 tablet_replica.cc:445] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c: This TxnStatusTablet replica's current role is: LEADER
W20260129 03:01:00.066035 27155 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:01:00.067090 27307 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260129 03:01:00.067615 27307 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260129 03:01:00.070932 27155 catalog_manager.cc:5487] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260129 03:01:00.097838 27255 tablet_service.cc:1505] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260129 03:01:00.099195 27255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:00.111778 27300 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c: Bootstrap starting.
I20260129 03:01:00.116973 27300 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c: Neither blocks nor log segments found. Creating new log.
I20260129 03:01:00.121512 27300 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c: No bootstrap required, opened a new log
I20260129 03:01:00.121989 27300 ts_tablet_manager.cc:1403] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.007s
I20260129 03:01:00.124030 27300 raft_consensus.cc:359] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "09a7f630b7864ad4a148402b48d1702c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 42195 } }
I20260129 03:01:00.124464 27300 raft_consensus.cc:385] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:01:00.124703 27300 raft_consensus.cc:740] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a7f630b7864ad4a148402b48d1702c, State: Initialized, Role: FOLLOWER
I20260129 03:01:00.125190 27300 consensus_queue.cc:260] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "09a7f630b7864ad4a148402b48d1702c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 42195 } }
I20260129 03:01:00.125613 27300 raft_consensus.cc:399] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:01:00.125859 27300 raft_consensus.cc:493] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:01:00.126104 27300 raft_consensus.cc:3060] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:01:00.131278 27300 raft_consensus.cc:515] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "09a7f630b7864ad4a148402b48d1702c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 42195 } }
I20260129 03:01:00.131814 27300 leader_election.cc:304] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09a7f630b7864ad4a148402b48d1702c; no voters:
I20260129 03:01:00.132277 27300 leader_election.cc:290] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:01:00.132463 27303 raft_consensus.cc:2804] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:01:00.132930 27303 raft_consensus.cc:697] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 1 LEADER]: Becoming Leader. State: Replica: 09a7f630b7864ad4a148402b48d1702c, State: Running, Role: LEADER
I20260129 03:01:00.133648 27303 consensus_queue.cc:237] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "09a7f630b7864ad4a148402b48d1702c" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 42195 } }
I20260129 03:01:00.134127 27300 ts_tablet_manager.cc:1434] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260129 03:01:00.139187 27155 catalog_manager.cc:5487] Ignoring report from unknown tablet 22222222222222222222222222222222
W20260129 03:01:00.142933 27155 catalog_manager.cc:5487] Ignoring report from unknown tablet 22222222222222222222222222222222
I20260129 03:01:00.162417 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:01:00.182114 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:01:00.183074 23477 tablet_replica.cc:333] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c: stopping tablet replica
I20260129 03:01:00.183707 23477 raft_consensus.cc:2243] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:01:00.184136 23477 raft_consensus.cc:2272] T 22222222222222222222222222222222 P 09a7f630b7864ad4a148402b48d1702c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:00.186705 23477 tablet_replica.cc:333] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c: stopping tablet replica
I20260129 03:01:00.187166 23477 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:01:00.187603 23477 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 09a7f630b7864ad4a148402b48d1702c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:00.208902 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:01:00.223863 23477 master.cc:561] Master@127.22.237.126:36137 shutting down...
I20260129 03:01:00.244040 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:01:00.244752 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:00.245249 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9242f521b30e47aa841c4e373b4d7965: stopping tablet replica
I20260129 03:01:00.265841 23477 master.cc:583] Master@127.22.237.126:36137 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (641 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (6406 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20260129 03:01:00.298285 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:38259
I20260129 03:01:00.299417 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:01:00.304999 27317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:00.305351 27318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:00.306963 27320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:00.307736 23477 server_base.cc:1060] running on GCE node
I20260129 03:01:00.308534 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:01:00.308727 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:01:00.308842 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655660308831 us; error 0 us; skew 500 ppm
I20260129 03:01:00.309320 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:01:00.311875 23477 webserver.cc:492] Webserver started at http://127.22.237.126:34109/ using document root <none> and password file <none>
I20260129 03:01:00.312317 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:01:00.312476 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:01:00.312695 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:01:00.313824 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "688e2ca469aa4929a72c830d325f9d3b"
format_stamp: "Formatted at 2026-01-29 03:01:00 on dist-test-slave-1zcf"
I20260129 03:01:00.318321 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260129 03:01:00.321882 27325 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:00.322625 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260129 03:01:00.322894 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "688e2ca469aa4929a72c830d325f9d3b"
format_stamp: "Formatted at 2026-01-29 03:01:00 on dist-test-slave-1zcf"
I20260129 03:01:00.323179 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:01:00.344172 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:01:00.345538 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:01:00.383172 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:38259
I20260129 03:01:00.383277 27376 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:38259 every 8 connection(s)
I20260129 03:01:00.387055 27377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:00.398254 27377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b: Bootstrap starting.
I20260129 03:01:00.403131 27377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b: Neither blocks nor log segments found. Creating new log.
I20260129 03:01:00.407342 27377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b: No bootstrap required, opened a new log
I20260129 03:01:00.409399 27377 raft_consensus.cc:359] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688e2ca469aa4929a72c830d325f9d3b" member_type: VOTER }
I20260129 03:01:00.409858 27377 raft_consensus.cc:385] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:01:00.410041 27377 raft_consensus.cc:740] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688e2ca469aa4929a72c830d325f9d3b, State: Initialized, Role: FOLLOWER
I20260129 03:01:00.410550 27377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688e2ca469aa4929a72c830d325f9d3b" member_type: VOTER }
I20260129 03:01:00.411001 27377 raft_consensus.cc:399] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:01:00.411196 27377 raft_consensus.cc:493] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:01:00.411398 27377 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:01:00.415928 27377 raft_consensus.cc:515] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688e2ca469aa4929a72c830d325f9d3b" member_type: VOTER }
I20260129 03:01:00.416446 27377 leader_election.cc:304] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 688e2ca469aa4929a72c830d325f9d3b; no voters:
I20260129 03:01:00.417555 27377 leader_election.cc:290] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:01:00.418008 27380 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:01:00.419333 27380 raft_consensus.cc:697] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 1 LEADER]: Becoming Leader. State: Replica: 688e2ca469aa4929a72c830d325f9d3b, State: Running, Role: LEADER
I20260129 03:01:00.420040 27380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688e2ca469aa4929a72c830d325f9d3b" member_type: VOTER }
I20260129 03:01:00.420420 27377 sys_catalog.cc:565] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:01:00.422751 27382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 688e2ca469aa4929a72c830d325f9d3b. Latest consensus state: current_term: 1 leader_uuid: "688e2ca469aa4929a72c830d325f9d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688e2ca469aa4929a72c830d325f9d3b" member_type: VOTER } }
I20260129 03:01:00.422796 27381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "688e2ca469aa4929a72c830d325f9d3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688e2ca469aa4929a72c830d325f9d3b" member_type: VOTER } }
I20260129 03:01:00.423486 27381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [sys.catalog]: This master's current role is: LEADER
I20260129 03:01:00.424089 27382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [sys.catalog]: This master's current role is: LEADER
I20260129 03:01:00.427307 27385 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:01:00.432286 27385 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:01:00.435382 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:01:00.440940 27385 catalog_manager.cc:1357] Generated new cluster ID: 044fb847839d498198e187d9eda555ab
I20260129 03:01:00.441265 27385 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:01:00.460528 27385 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:01:00.461892 27385 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:01:00.477324 27385 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b: Generated new TSK 0
I20260129 03:01:00.477988 27385 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:01:00.502403 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:01:00.508152 27398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:00.509068 27399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:00.510666 27401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:00.512082 23477 server_base.cc:1060] running on GCE node
I20260129 03:01:00.512836 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:01:00.513027 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:01:00.513146 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655660513135 us; error 0 us; skew 500 ppm
I20260129 03:01:00.513609 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:01:00.515863 23477 webserver.cc:492] Webserver started at http://127.22.237.65:46159/ using document root <none> and password file <none>
I20260129 03:01:00.516312 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:01:00.516461 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:01:00.516669 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:01:00.517802 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "45555c14df8244cdb87ea87a01727ee7"
format_stamp: "Formatted at 2026-01-29 03:01:00 on dist-test-slave-1zcf"
I20260129 03:01:00.522313 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260129 03:01:00.525923 27406 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:00.526700 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260129 03:01:00.526983 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "45555c14df8244cdb87ea87a01727ee7"
format_stamp: "Formatted at 2026-01-29 03:01:00 on dist-test-slave-1zcf"
I20260129 03:01:00.527256 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:01:00.547362 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:01:00.548557 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:01:00.550047 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:01:00.552567 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:01:00.552778 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:00.552999 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:01:00.553148 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:00.593180 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:43775
I20260129 03:01:00.593286 27468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:43775 every 8 connection(s)
I20260129 03:01:00.597800 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:01:00.605262 27473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:00.605887 27474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:00.611737 27476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:00.611797 27469 heartbeater.cc:344] Connected to a master server at 127.22.237.126:38259
I20260129 03:01:00.612368 27469 heartbeater.cc:461] Registering TS with master...
I20260129 03:01:00.612537 23477 server_base.cc:1060] running on GCE node
I20260129 03:01:00.613302 27469 heartbeater.cc:507] Master 127.22.237.126:38259 requested a full tablet report, sending...
I20260129 03:01:00.613596 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:01:00.613971 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:01:00.614168 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655660614151 us; error 0 us; skew 500 ppm
I20260129 03:01:00.614878 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:01:00.615521 27342 ts_manager.cc:194] Registered new tserver with Master: 45555c14df8244cdb87ea87a01727ee7 (127.22.237.65:43775)
I20260129 03:01:00.617429 23477 webserver.cc:492] Webserver started at http://127.22.237.66:38031/ using document root <none> and password file <none>
I20260129 03:01:00.617808 27342 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51996
I20260129 03:01:00.617974 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:01:00.618235 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:01:00.618507 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:01:00.620023 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-1-root/instance:
uuid: "4e087682bdac493e97015a7e2c3b38f5"
format_stamp: "Formatted at 2026-01-29 03:01:00 on dist-test-slave-1zcf"
I20260129 03:01:00.624663 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260129 03:01:00.628100 27481 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:00.628893 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 03:01:00.629223 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-1-root
uuid: "4e087682bdac493e97015a7e2c3b38f5"
format_stamp: "Formatted at 2026-01-29 03:01:00 on dist-test-slave-1zcf"
I20260129 03:01:00.629485 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:01:00.644716 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:01:00.645918 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:01:00.647301 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:01:00.649432 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:01:00.649610 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:00.649883 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:01:00.650027 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:00.687057 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.66:40725
I20260129 03:01:00.687151 27543 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.66:40725 every 8 connection(s)
I20260129 03:01:00.691690 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:01:00.698956 27547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:00.700982 27548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:00.702121 27550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:00.703297 27544 heartbeater.cc:344] Connected to a master server at 127.22.237.126:38259
I20260129 03:01:00.703497 23477 server_base.cc:1060] running on GCE node
I20260129 03:01:00.703786 27544 heartbeater.cc:461] Registering TS with master...
I20260129 03:01:00.704465 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:01:00.704777 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:01:00.704684 27544 heartbeater.cc:507] Master 127.22.237.126:38259 requested a full tablet report, sending...
I20260129 03:01:00.705072 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655660705053 us; error 0 us; skew 500 ppm
I20260129 03:01:00.705870 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:01:00.707161 27342 ts_manager.cc:194] Registered new tserver with Master: 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725)
I20260129 03:01:00.708658 23477 webserver.cc:492] Webserver started at http://127.22.237.67:45995/ using document root <none> and password file <none>
I20260129 03:01:00.709043 27342 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52004
I20260129 03:01:00.709316 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:01:00.709565 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:01:00.709925 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:01:00.711367 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-2-root/instance:
uuid: "687638b2fa0e4c029e7e07fa02fb9dbe"
format_stamp: "Formatted at 2026-01-29 03:01:00 on dist-test-slave-1zcf"
I20260129 03:01:00.716050 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 03:01:00.719594 27555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:00.720396 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260129 03:01:00.720698 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-2-root
uuid: "687638b2fa0e4c029e7e07fa02fb9dbe"
format_stamp: "Formatted at 2026-01-29 03:01:00 on dist-test-slave-1zcf"
I20260129 03:01:00.720958 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:01:00.747375 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:01:00.748495 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:01:00.750003 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:01:00.752154 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:01:00.752327 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:00.752547 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:01:00.752687 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:00.792946 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.67:41429
I20260129 03:01:00.793038 27617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.67:41429 every 8 connection(s)
I20260129 03:01:00.805737 27618 heartbeater.cc:344] Connected to a master server at 127.22.237.126:38259
I20260129 03:01:00.806107 27618 heartbeater.cc:461] Registering TS with master...
I20260129 03:01:00.806823 27618 heartbeater.cc:507] Master 127.22.237.126:38259 requested a full tablet report, sending...
I20260129 03:01:00.808517 27342 ts_manager.cc:194] Registered new tserver with Master: 687638b2fa0e4c029e7e07fa02fb9dbe (127.22.237.67:41429)
I20260129 03:01:00.809355 23477 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013270727s
I20260129 03:01:00.809880 27342 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52008
I20260129 03:01:00.850566 27342 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52030:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 03:01:00.905740 27583 tablet_service.cc:1505] Processing CreateTablet for tablet caa8ae3fcb2145ffa95e17e222778e09 (DEFAULT_TABLE table=test-workload [id=4e6b12c06e9047a6a9999c01a6f45fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 03:01:00.907187 27583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa8ae3fcb2145ffa95e17e222778e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:00.907488 27434 tablet_service.cc:1505] Processing CreateTablet for tablet caa8ae3fcb2145ffa95e17e222778e09 (DEFAULT_TABLE table=test-workload [id=4e6b12c06e9047a6a9999c01a6f45fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 03:01:00.908725 27434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa8ae3fcb2145ffa95e17e222778e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:00.908833 27509 tablet_service.cc:1505] Processing CreateTablet for tablet caa8ae3fcb2145ffa95e17e222778e09 (DEFAULT_TABLE table=test-workload [id=4e6b12c06e9047a6a9999c01a6f45fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 03:01:00.909898 27509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa8ae3fcb2145ffa95e17e222778e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:00.925631 27640 tablet_bootstrap.cc:492] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7: Bootstrap starting.
I20260129 03:01:00.927673 27638 tablet_bootstrap.cc:492] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Bootstrap starting.
I20260129 03:01:00.929600 27639 tablet_bootstrap.cc:492] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe: Bootstrap starting.
I20260129 03:01:00.934397 27640 tablet_bootstrap.cc:654] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7: Neither blocks nor log segments found. Creating new log.
I20260129 03:01:00.935118 27638 tablet_bootstrap.cc:654] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Neither blocks nor log segments found. Creating new log.
I20260129 03:01:00.936631 27639 tablet_bootstrap.cc:654] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe: Neither blocks nor log segments found. Creating new log.
I20260129 03:01:00.944247 27640 tablet_bootstrap.cc:492] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7: No bootstrap required, opened a new log
I20260129 03:01:00.944814 27640 ts_tablet_manager.cc:1403] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20260129 03:01:00.945127 27638 tablet_bootstrap.cc:492] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: No bootstrap required, opened a new log
I20260129 03:01:00.945188 27639 tablet_bootstrap.cc:492] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe: No bootstrap required, opened a new log
I20260129 03:01:00.945771 27638 ts_tablet_manager.cc:1403] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20260129 03:01:00.945847 27639 ts_tablet_manager.cc:1403] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20260129 03:01:00.947708 27640 raft_consensus.cc:359] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:00.948170 27639 raft_consensus.cc:359] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:00.948482 27640 raft_consensus.cc:385] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:01:00.948779 27639 raft_consensus.cc:385] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:01:00.948849 27640 raft_consensus.cc:740] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45555c14df8244cdb87ea87a01727ee7, State: Initialized, Role: FOLLOWER
I20260129 03:01:00.949195 27639 raft_consensus.cc:740] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687638b2fa0e4c029e7e07fa02fb9dbe, State: Initialized, Role: FOLLOWER
I20260129 03:01:00.948789 27638 raft_consensus.cc:359] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:00.949904 27638 raft_consensus.cc:385] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:01:00.950387 27638 raft_consensus.cc:740] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e087682bdac493e97015a7e2c3b38f5, State: Initialized, Role: FOLLOWER
I20260129 03:01:00.949946 27639 consensus_queue.cc:260] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:00.949916 27640 consensus_queue.cc:260] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:00.951284 27638 consensus_queue.cc:260] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:00.957049 27544 heartbeater.cc:499] Master 127.22.237.126:38259 was elected leader, sending a full tablet report...
I20260129 03:01:00.958336 27638 ts_tablet_manager.cc:1434] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260129 03:01:00.959791 27640 ts_tablet_manager.cc:1434] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260129 03:01:00.960333 27618 heartbeater.cc:499] Master 127.22.237.126:38259 was elected leader, sending a full tablet report...
I20260129 03:01:00.960544 27469 heartbeater.cc:499] Master 127.22.237.126:38259 was elected leader, sending a full tablet report...
I20260129 03:01:00.960966 27639 ts_tablet_manager.cc:1434] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe: Time spent starting tablet: real 0.015s user 0.005s sys 0.008s
I20260129 03:01:00.975106 27645 raft_consensus.cc:493] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:01:00.975567 27645 raft_consensus.cc:515] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:00.977483 27645 leader_election.cc:290] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 687638b2fa0e4c029e7e07fa02fb9dbe (127.22.237.67:41429), 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725)
I20260129 03:01:00.986202 27593 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caa8ae3fcb2145ffa95e17e222778e09" candidate_uuid: "45555c14df8244cdb87ea87a01727ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" is_pre_election: true
I20260129 03:01:00.987047 27593 raft_consensus.cc:2468] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45555c14df8244cdb87ea87a01727ee7 in term 0.
I20260129 03:01:00.988032 27519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caa8ae3fcb2145ffa95e17e222778e09" candidate_uuid: "45555c14df8244cdb87ea87a01727ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e087682bdac493e97015a7e2c3b38f5" is_pre_election: true
I20260129 03:01:00.988204 27407 leader_election.cc:304] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45555c14df8244cdb87ea87a01727ee7, 687638b2fa0e4c029e7e07fa02fb9dbe; no voters:
I20260129 03:01:00.988567 27519 raft_consensus.cc:2468] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45555c14df8244cdb87ea87a01727ee7 in term 0.
I20260129 03:01:00.988981 27645 raft_consensus.cc:2804] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 03:01:00.989323 27645 raft_consensus.cc:493] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:01:00.989651 27645 raft_consensus.cc:3060] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:01:00.994516 27645 raft_consensus.cc:515] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:00.996017 27645 leader_election.cc:290] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [CANDIDATE]: Term 1 election: Requested vote from peers 687638b2fa0e4c029e7e07fa02fb9dbe (127.22.237.67:41429), 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725)
I20260129 03:01:00.996886 27593 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caa8ae3fcb2145ffa95e17e222778e09" candidate_uuid: "45555c14df8244cdb87ea87a01727ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe"
I20260129 03:01:00.997126 27519 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caa8ae3fcb2145ffa95e17e222778e09" candidate_uuid: "45555c14df8244cdb87ea87a01727ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e087682bdac493e97015a7e2c3b38f5"
I20260129 03:01:00.997370 27593 raft_consensus.cc:3060] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:01:00.997751 27519 raft_consensus.cc:3060] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:01:01.002404 27593 raft_consensus.cc:2468] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45555c14df8244cdb87ea87a01727ee7 in term 1.
I20260129 03:01:01.002771 27519 raft_consensus.cc:2468] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45555c14df8244cdb87ea87a01727ee7 in term 1.
I20260129 03:01:01.003419 27407 leader_election.cc:304] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45555c14df8244cdb87ea87a01727ee7, 687638b2fa0e4c029e7e07fa02fb9dbe; no voters:
I20260129 03:01:01.004144 27645 raft_consensus.cc:2804] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:01:01.005031 27645 raft_consensus.cc:697] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 1 LEADER]: Becoming Leader. State: Replica: 45555c14df8244cdb87ea87a01727ee7, State: Running, Role: LEADER
I20260129 03:01:01.005638 27645 consensus_queue.cc:237] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:01.013558 27342 catalog_manager.cc:5654] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45555c14df8244cdb87ea87a01727ee7 (127.22.237.65). New cstate: current_term: 1 leader_uuid: "45555c14df8244cdb87ea87a01727ee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } }
I20260129 03:01:01.263240 27434 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260129 03:01:01.264098 27509 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:01.265198 27583 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
688e2ca469aa4929a72c830d325f9d3b | 127.22.237.126:38259 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
45555c14df8244cdb87ea87a01727ee7 | 127.22.237.65:43775 | HEALTHY | <none> | 1 | 0
4e087682bdac493e97015a7e2c3b38f5 | 127.22.237.66:40725 | HEALTHY | <none> | 0 | 0
687638b2fa0e4c029e7e07fa02fb9dbe | 127.22.237.67:41429 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet caa8ae3fcb2145ffa95e17e222778e09 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
687638b2fa0e4c029e7e07fa02fb9dbe (127.22.237.67:41429): RUNNING
4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725): RUNNING
45555c14df8244cdb87ea87a01727ee7 (127.22.237.65:43775): RUNNING [LEADER]
All reported replicas are:
A = 687638b2fa0e4c029e7e07fa02fb9dbe
B = 4e087682bdac493e97015a7e2c3b38f5
C = 45555c14df8244cdb87ea87a01727ee7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
I20260129 03:01:01.487946 27645 consensus_queue.cc:1048] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260129 03:01:01.498790 23477 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260129 03:01:01.505869 27648 consensus_queue.cc:1048] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:01:01.866506 27434 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260129 03:01:01.867265 27509 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:01.872193 27583 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
688e2ca469aa4929a72c830d325f9d3b | 127.22.237.126:38259 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
45555c14df8244cdb87ea87a01727ee7 | 127.22.237.65:43775 | HEALTHY | <none> | 1 | 0
4e087682bdac493e97015a7e2c3b38f5 | 127.22.237.66:40725 | HEALTHY | <none> | 0 | 0
687638b2fa0e4c029e7e07fa02fb9dbe | 127.22.237.67:41429 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260129 03:01:02.075346 23477 log_verifier.cc:126] Checking tablet caa8ae3fcb2145ffa95e17e222778e09
I20260129 03:01:02.137161 23477 log_verifier.cc:177] Verified matching terms for 1 ops in tablet caa8ae3fcb2145ffa95e17e222778e09
I20260129 03:01:02.137781 23477 tablet_replica.cc:333] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: stopping tablet replica
I20260129 03:01:02.138293 23477 raft_consensus.cc:2243] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 03:01:02.138761 23477 raft_consensus.cc:2272] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:02.343175 27434 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260129 03:01:02.344539 27509 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:02.346737 27583 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
W20260129 03:01:02.455691 27408 consensus_peers.cc:597] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 -> Peer 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725): Couldn't send request to peer 4e087682bdac493e97015a7e2c3b38f5. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260129 03:01:02.469838 27648 raft_consensus.cc:1012] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7: Attempting to remove follower 4e087682bdac493e97015a7e2c3b38f5 from the Raft config. Reason: The tablet replica hosted on peer 4e087682bdac493e97015a7e2c3b38f5 has failed
I20260129 03:01:02.472054 27648 consensus_queue.cc:237] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:02.476441 27593 raft_consensus.cc:1275] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Refusing update from remote peer 45555c14df8244cdb87ea87a01727ee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 03:01:02.477914 27648 consensus_queue.cc:1048] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 03:01:02.484771 27732 raft_consensus.cc:2955] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } }
I20260129 03:01:02.486712 27593 raft_consensus.cc:2955] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } }
I20260129 03:01:02.498584 27341 catalog_manager.cc:5654] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 reported cstate change: config changed from index -1 to 2, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) evicted. New cstate: current_term: 1 leader_uuid: "45555c14df8244cdb87ea87a01727ee7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } }
I20260129 03:01:02.520653 27509 tablet_service.cc:1552] Processing DeleteTablet for tablet caa8ae3fcb2145ffa95e17e222778e09 with delete_type TABLET_DATA_TOMBSTONED (TS 4e087682bdac493e97015a7e2c3b38f5 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:59572
I20260129 03:01:02.524899 27444 consensus_queue.cc:237] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } }
W20260129 03:01:02.533708 27408 consensus_peers.cc:597] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 -> Peer 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725): Couldn't send request to peer 4e087682bdac493e97015a7e2c3b38f5. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260129 03:01:02.536446 27593 raft_consensus.cc:1275] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Refusing update from remote peer 45555c14df8244cdb87ea87a01727ee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 03:01:02.538100 27732 consensus_queue.cc:1048] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:01:02.538409 27738 ts_tablet_manager.cc:1916] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260129 03:01:02.543398 27648 raft_consensus.cc:2955] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } }
I20260129 03:01:02.545336 27593 raft_consensus.cc:2955] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } }
I20260129 03:01:02.554037 27328 catalog_manager.cc:5167] ChangeConfig:ADD_PEER:VOTER RPC for tablet caa8ae3fcb2145ffa95e17e222778e09 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260129 03:01:02.554801 27738 ts_tablet_manager.cc:1929] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260129 03:01:02.555375 27738 log.cc:1199] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Deleting WAL directory at /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1769655599110862-23477-0/minicluster-data/ts-1-root/wals/caa8ae3fcb2145ffa95e17e222778e09
I20260129 03:01:02.557068 27327 catalog_manager.cc:4985] TS 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725): tablet caa8ae3fcb2145ffa95e17e222778e09 (table test-workload [id=4e6b12c06e9047a6a9999c01a6f45fe4]) successfully deleted
I20260129 03:01:02.562083 27341 catalog_manager.cc:5654] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 reported cstate change: config changed from index 2 to 3, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) added. New cstate: current_term: 1 leader_uuid: "45555c14df8244cdb87ea87a01727ee7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
688e2ca469aa4929a72c830d325f9d3b | 127.22.237.126:38259 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
45555c14df8244cdb87ea87a01727ee7 | 127.22.237.65:43775 | HEALTHY | <none> | 1 | 0
4e087682bdac493e97015a7e2c3b38f5 | 127.22.237.66:40725 | HEALTHY | <none> | 0 | 0
687638b2fa0e4c029e7e07fa02fb9dbe | 127.22.237.67:41429 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet caa8ae3fcb2145ffa95e17e222778e09 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
687638b2fa0e4c029e7e07fa02fb9dbe (127.22.237.67:41429): RUNNING
4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
45555c14df8244cdb87ea87a01727ee7 (127.22.237.65:43775): RUNNING [LEADER]
All reported replicas are:
A = 687638b2fa0e4c029e7e07fa02fb9dbe
B = 4e087682bdac493e97015a7e2c3b38f5
C = 45555c14df8244cdb87ea87a01727ee7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260129 03:01:02.572046 23477 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260129 03:01:02.924614 27434 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260129 03:01:02.926529 27509 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:02.935328 27583 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:02.954686 27648 raft_consensus.cc:1012] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7: Attempting to remove follower 4e087682bdac493e97015a7e2c3b38f5 from the Raft config. Reason: The tablet replica hosted on peer 4e087682bdac493e97015a7e2c3b38f5 has failed
I20260129 03:01:02.956463 27648 consensus_queue.cc:237] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } }
I20260129 03:01:02.960642 27593 raft_consensus.cc:1275] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Refusing update from remote peer 45555c14df8244cdb87ea87a01727ee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260129 03:01:02.962216 27732 consensus_queue.cc:1048] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260129 03:01:02.968954 27648 raft_consensus.cc:2955] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } }
I20260129 03:01:02.981833 27593 raft_consensus.cc:2955] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } }
I20260129 03:01:02.996351 27342 catalog_manager.cc:5654] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe reported cstate change: config changed from index 3 to 4, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) evicted. New cstate: current_term: 1 leader_uuid: "45555c14df8244cdb87ea87a01727ee7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } }
I20260129 03:01:03.014467 27509 tablet_service.cc:1552] Processing DeleteTablet for tablet caa8ae3fcb2145ffa95e17e222778e09 with delete_type TABLET_DATA_TOMBSTONED (TS 4e087682bdac493e97015a7e2c3b38f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59572
I20260129 03:01:03.015969 27327 catalog_manager.cc:4985] TS 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725): tablet caa8ae3fcb2145ffa95e17e222778e09 (table test-workload [id=4e6b12c06e9047a6a9999c01a6f45fe4]) successfully deleted
I20260129 03:01:03.020125 27444 consensus_queue.cc:237] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } }
W20260129 03:01:03.027496 27408 consensus_peers.cc:597] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 -> Peer 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725): Couldn't send request to peer 4e087682bdac493e97015a7e2c3b38f5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260129 03:01:03.030807 27593 raft_consensus.cc:1275] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Refusing update from remote peer 45555c14df8244cdb87ea87a01727ee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260129 03:01:03.032963 27648 consensus_queue.cc:1048] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260129 03:01:03.040086 27732 raft_consensus.cc:2955] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } }
I20260129 03:01:03.045156 27593 raft_consensus.cc:2955] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } }
I20260129 03:01:03.061520 27328 catalog_manager.cc:5167] ChangeConfig:ADD_PEER:VOTER RPC for tablet caa8ae3fcb2145ffa95e17e222778e09 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260129 03:01:03.066988 27342 catalog_manager.cc:5654] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 reported cstate change: config changed from index 4 to 5, VOTER 4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66) added. New cstate: current_term: 1 leader_uuid: "45555c14df8244cdb87ea87a01727ee7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
688e2ca469aa4929a72c830d325f9d3b | 127.22.237.126:38259 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
45555c14df8244cdb87ea87a01727ee7 | 127.22.237.65:43775 | HEALTHY | <none> | 1 | 0
4e087682bdac493e97015a7e2c3b38f5 | 127.22.237.66:40725 | HEALTHY | <none> | 0 | 0
687638b2fa0e4c029e7e07fa02fb9dbe | 127.22.237.67:41429 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet caa8ae3fcb2145ffa95e17e222778e09 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
687638b2fa0e4c029e7e07fa02fb9dbe (127.22.237.67:41429): RUNNING
45555c14df8244cdb87ea87a01727ee7 (127.22.237.65:43775): RUNNING [LEADER]
4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 687638b2fa0e4c029e7e07fa02fb9dbe
B = 45555c14df8244cdb87ea87a01727ee7
C = 4e087682bdac493e97015a7e2c3b38f5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 3 | Yes
B | A B* C | 1 | 3 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260129 03:01:03.153359 23477 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260129 03:01:03.506404 27773 ts_tablet_manager.cc:933] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Initiating tablet copy from peer 45555c14df8244cdb87ea87a01727ee7 (127.22.237.65:43775)
I20260129 03:01:03.508204 27773 tablet_copy_client.cc:323] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.237.65:43775
I20260129 03:01:03.521749 27454 tablet_copy_service.cc:140] P 45555c14df8244cdb87ea87a01727ee7: Received BeginTabletCopySession request for tablet caa8ae3fcb2145ffa95e17e222778e09 from peer 4e087682bdac493e97015a7e2c3b38f5 ({username='slave'} at 127.0.0.1:48610)
I20260129 03:01:03.522359 27454 tablet_copy_service.cc:161] P 45555c14df8244cdb87ea87a01727ee7: Beginning new tablet copy session on tablet caa8ae3fcb2145ffa95e17e222778e09 from peer 4e087682bdac493e97015a7e2c3b38f5 at {username='slave'} at 127.0.0.1:48610: session id = 4e087682bdac493e97015a7e2c3b38f5-caa8ae3fcb2145ffa95e17e222778e09
I20260129 03:01:03.529985 27454 tablet_copy_source_session.cc:215] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 03:01:03.532577 27773 ts_tablet_manager.cc:1916] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260129 03:01:03.545331 27773 ts_tablet_manager.cc:1929] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260129 03:01:03.546136 27773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa8ae3fcb2145ffa95e17e222778e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:03.552788 27773 tablet_copy_client.cc:806] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: tablet copy: Starting download of 0 data blocks...
I20260129 03:01:03.553310 27773 tablet_copy_client.cc:670] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: tablet copy: Starting download of 1 WAL segments...
I20260129 03:01:03.556777 27773 tablet_copy_client.cc:538] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 03:01:03.564250 27773 tablet_bootstrap.cc:492] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Bootstrap starting.
I20260129 03:01:03.593886 27434 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260129 03:01:03.598883 27509 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:03.610674 27773 tablet_bootstrap.cc:492] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 03:01:03.611613 27773 tablet_bootstrap.cc:492] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Bootstrap complete.
I20260129 03:01:03.612051 27583 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:03.612443 27773 ts_tablet_manager.cc:1403] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Time spent bootstrapping tablet: real 0.048s user 0.040s sys 0.008s
I20260129 03:01:03.615437 27773 raft_consensus.cc:359] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } }
I20260129 03:01:03.616056 27773 raft_consensus.cc:740] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e087682bdac493e97015a7e2c3b38f5, State: Initialized, Role: FOLLOWER
I20260129 03:01:03.616743 27773 consensus_queue.cc:260] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "687638b2fa0e4c029e7e07fa02fb9dbe" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 41429 } } peers { permanent_uuid: "45555c14df8244cdb87ea87a01727ee7" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 43775 } } peers { permanent_uuid: "4e087682bdac493e97015a7e2c3b38f5" member_type: VOTER last_known_addr { host: "127.22.237.66" port: 40725 } }
I20260129 03:01:03.619170 27773 ts_tablet_manager.cc:1434] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260129 03:01:03.620987 27454 tablet_copy_service.cc:342] P 45555c14df8244cdb87ea87a01727ee7: Request end of tablet copy session 4e087682bdac493e97015a7e2c3b38f5-caa8ae3fcb2145ffa95e17e222778e09 received from {username='slave'} at 127.0.0.1:48610
I20260129 03:01:03.621364 27454 tablet_copy_service.cc:434] P 45555c14df8244cdb87ea87a01727ee7: ending tablet copy session 4e087682bdac493e97015a7e2c3b38f5-caa8ae3fcb2145ffa95e17e222778e09 on tablet caa8ae3fcb2145ffa95e17e222778e09 with peer 4e087682bdac493e97015a7e2c3b38f5
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
688e2ca469aa4929a72c830d325f9d3b | 127.22.237.126:38259 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
45555c14df8244cdb87ea87a01727ee7 | 127.22.237.65:43775 | HEALTHY | <none> | 1 | 0
4e087682bdac493e97015a7e2c3b38f5 | 127.22.237.66:40725 | HEALTHY | <none> | 0 | 0
687638b2fa0e4c029e7e07fa02fb9dbe | 127.22.237.67:41429 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet caa8ae3fcb2145ffa95e17e222778e09 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
687638b2fa0e4c029e7e07fa02fb9dbe (127.22.237.67:41429): RUNNING
45555c14df8244cdb87ea87a01727ee7 (127.22.237.65:43775): RUNNING [LEADER]
4e087682bdac493e97015a7e2c3b38f5 (127.22.237.66:40725): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap starting.
All reported replicas are:
A = 687638b2fa0e4c029e7e07fa02fb9dbe
B = 45555c14df8244cdb87ea87a01727ee7
C = 4e087682bdac493e97015a7e2c3b38f5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 5 | Yes
B | A B* C | 1 | 5 | Yes
C | A B C | 1 | 5 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260129 03:01:03.798020 23477 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260129 03:01:03.977231 27519 raft_consensus.cc:1217] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260129 03:01:04.339048 27509 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:04.339084 27434 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260129 03:01:04.342657 27583 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
688e2ca469aa4929a72c830d325f9d3b | 127.22.237.126:38259 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
45555c14df8244cdb87ea87a01727ee7 | 127.22.237.65:43775 | HEALTHY | <none> | 1 | 0
4e087682bdac493e97015a7e2c3b38f5 | 127.22.237.66:40725 | HEALTHY | <none> | 0 | 0
687638b2fa0e4c029e7e07fa02fb9dbe | 127.22.237.67:41429 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260129 03:01:05.069151 23477 log_verifier.cc:126] Checking tablet caa8ae3fcb2145ffa95e17e222778e09
I20260129 03:01:05.126772 23477 log_verifier.cc:177] Verified matching terms for 5 ops in tablet caa8ae3fcb2145ffa95e17e222778e09
I20260129 03:01:05.128721 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:01:05.149154 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:01:05.149796 23477 tablet_replica.cc:333] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7: stopping tablet replica
I20260129 03:01:05.150264 23477 raft_consensus.cc:2243] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:01:05.151002 23477 raft_consensus.cc:2272] T caa8ae3fcb2145ffa95e17e222778e09 P 45555c14df8244cdb87ea87a01727ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:05.171150 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:01:05.180303 23477 tablet_server.cc:178] TabletServer@127.22.237.66:0 shutting down...
I20260129 03:01:05.203136 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:01:05.203725 23477 tablet_replica.cc:333] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5: stopping tablet replica
I20260129 03:01:05.204201 23477 raft_consensus.cc:2243] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 03:01:05.204566 23477 raft_consensus.cc:2272] T caa8ae3fcb2145ffa95e17e222778e09 P 4e087682bdac493e97015a7e2c3b38f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:05.223461 23477 tablet_server.cc:195] TabletServer@127.22.237.66:0 shutdown complete.
I20260129 03:01:05.232497 23477 tablet_server.cc:178] TabletServer@127.22.237.67:0 shutting down...
I20260129 03:01:05.249976 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:01:05.250547 23477 tablet_replica.cc:333] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe: stopping tablet replica
I20260129 03:01:05.251056 23477 raft_consensus.cc:2243] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 03:01:05.251487 23477 raft_consensus.cc:2272] T caa8ae3fcb2145ffa95e17e222778e09 P 687638b2fa0e4c029e7e07fa02fb9dbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:05.270738 23477 tablet_server.cc:195] TabletServer@127.22.237.67:0 shutdown complete.
I20260129 03:01:05.279276 23477 master.cc:561] Master@127.22.237.126:38259 shutting down...
I20260129 03:01:05.294066 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:01:05.294575 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:05.294881 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 688e2ca469aa4929a72c830d325f9d3b: stopping tablet replica
W20260129 03:01:05.486261 27373 debug-util.cc:398] Leaking SignalData structure 0x7b0800062f40 after lost signal to thread 23480
W20260129 03:01:06.310642 23477 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 27373)
W20260129 03:01:07.311102 23477 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 27373)
I20260129 03:01:08.211781 23477 master.cc:583] Master@127.22.237.126:38259 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (7942 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20260129 03:01:08.240372 23477 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.22.237.126:45981
I20260129 03:01:08.241354 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:01:08.246224 27829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:08.247189 27830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:08.249545 27832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:08.249589 23477 server_base.cc:1060] running on GCE node
I20260129 03:01:08.251652 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:01:08.251870 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:01:08.252012 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655668251994 us; error 0 us; skew 500 ppm
I20260129 03:01:08.252486 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:01:08.254870 23477 webserver.cc:492] Webserver started at http://127.22.237.126:39569/ using document root <none> and password file <none>
I20260129 03:01:08.255316 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:01:08.255483 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:01:08.255719 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:01:08.256742 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/master-0-root/instance:
uuid: "3957c37ca90e4228a0b77e60c9f742df"
format_stamp: "Formatted at 2026-01-29 03:01:08 on dist-test-slave-1zcf"
I20260129 03:01:08.260954 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260129 03:01:08.264328 27837 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.265156 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260129 03:01:08.265443 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/master-0-root
uuid: "3957c37ca90e4228a0b77e60c9f742df"
format_stamp: "Formatted at 2026-01-29 03:01:08 on dist-test-slave-1zcf"
I20260129 03:01:08.265758 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:01:08.276080 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:01:08.277021 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:01:08.311059 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.126:45981
I20260129 03:01:08.311167 27888 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.126:45981 every 8 connection(s)
I20260129 03:01:08.314635 27889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:08.325404 27889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df: Bootstrap starting.
I20260129 03:01:08.329864 27889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df: Neither blocks nor log segments found. Creating new log.
I20260129 03:01:08.333835 27889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df: No bootstrap required, opened a new log
I20260129 03:01:08.335839 27889 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3957c37ca90e4228a0b77e60c9f742df" member_type: VOTER }
I20260129 03:01:08.336232 27889 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:01:08.336465 27889 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3957c37ca90e4228a0b77e60c9f742df, State: Initialized, Role: FOLLOWER
I20260129 03:01:08.337023 27889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3957c37ca90e4228a0b77e60c9f742df" member_type: VOTER }
I20260129 03:01:08.337473 27889 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260129 03:01:08.337704 27889 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260129 03:01:08.338033 27889 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:01:08.342816 27889 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3957c37ca90e4228a0b77e60c9f742df" member_type: VOTER }
I20260129 03:01:08.343389 27889 leader_election.cc:304] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3957c37ca90e4228a0b77e60c9f742df; no voters:
I20260129 03:01:08.344504 27889 leader_election.cc:290] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260129 03:01:08.344852 27892 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:01:08.346100 27892 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 1 LEADER]: Becoming Leader. State: Replica: 3957c37ca90e4228a0b77e60c9f742df, State: Running, Role: LEADER
I20260129 03:01:08.346732 27892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3957c37ca90e4228a0b77e60c9f742df" member_type: VOTER }
I20260129 03:01:08.347394 27889 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [sys.catalog]: configured and running, proceeding with master startup.
I20260129 03:01:08.352339 27893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3957c37ca90e4228a0b77e60c9f742df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3957c37ca90e4228a0b77e60c9f742df" member_type: VOTER } }
I20260129 03:01:08.352496 27894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3957c37ca90e4228a0b77e60c9f742df. Latest consensus state: current_term: 1 leader_uuid: "3957c37ca90e4228a0b77e60c9f742df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3957c37ca90e4228a0b77e60c9f742df" member_type: VOTER } }
I20260129 03:01:08.352968 27893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [sys.catalog]: This master's current role is: LEADER
I20260129 03:01:08.353178 27894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [sys.catalog]: This master's current role is: LEADER
I20260129 03:01:08.354651 27900 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260129 03:01:08.359237 27900 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260129 03:01:08.362038 23477 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260129 03:01:08.368093 27900 catalog_manager.cc:1357] Generated new cluster ID: a8a0ce72be8546ccac0408dd38432aef
I20260129 03:01:08.368351 27900 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260129 03:01:08.389920 27900 catalog_manager.cc:1380] Generated new certificate authority record
I20260129 03:01:08.391220 27900 catalog_manager.cc:1514] Loading token signing keys...
I20260129 03:01:08.404147 27900 catalog_manager.cc:6027] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df: Generated new TSK 0
I20260129 03:01:08.404734 27900 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260129 03:01:08.429013 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:01:08.434357 27910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:08.435599 27911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:08.437311 27913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:08.437795 23477 server_base.cc:1060] running on GCE node
I20260129 03:01:08.438586 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:01:08.438797 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:01:08.438946 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655668438929 us; error 0 us; skew 500 ppm
I20260129 03:01:08.439472 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:01:08.441588 23477 webserver.cc:492] Webserver started at http://127.22.237.65:46263/ using document root <none> and password file <none>
I20260129 03:01:08.442087 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:01:08.442258 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:01:08.442487 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:01:08.443475 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-0-root/instance:
uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08"
format_stamp: "Formatted at 2026-01-29 03:01:08 on dist-test-slave-1zcf"
I20260129 03:01:08.447677 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260129 03:01:08.450897 27918 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.451587 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260129 03:01:08.451875 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-0-root
uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08"
format_stamp: "Formatted at 2026-01-29 03:01:08 on dist-test-slave-1zcf"
I20260129 03:01:08.452132 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:01:08.465904 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:01:08.466984 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:01:08.468346 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:01:08.470494 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:01:08.470706 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.470934 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:01:08.471082 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.507356 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.65:34501
I20260129 03:01:08.507457 27980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.65:34501 every 8 connection(s)
I20260129 03:01:08.511886 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:01:08.518493 27985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:08.519311 27986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:08.522993 23477 server_base.cc:1060] running on GCE node
W20260129 03:01:08.522910 27988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:08.524065 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:01:08.524264 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:01:08.524420 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655668524403 us; error 0 us; skew 500 ppm
I20260129 03:01:08.525092 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:01:08.527637 23477 webserver.cc:492] Webserver started at http://127.22.237.66:44589/ using document root <none> and password file <none>
I20260129 03:01:08.527872 27981 heartbeater.cc:344] Connected to a master server at 127.22.237.126:45981
I20260129 03:01:08.528180 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:01:08.528213 27981 heartbeater.cc:461] Registering TS with master...
I20260129 03:01:08.528525 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:01:08.528811 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:01:08.529101 27981 heartbeater.cc:507] Master 127.22.237.126:45981 requested a full tablet report, sending...
I20260129 03:01:08.529922 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-1-root/instance:
uuid: "20f7ff56a6a5447899f5d9025961eaf2"
format_stamp: "Formatted at 2026-01-29 03:01:08 on dist-test-slave-1zcf"
I20260129 03:01:08.531226 27854 ts_manager.cc:194] Registered new tserver with Master: 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 (127.22.237.65:34501)
I20260129 03:01:08.533594 27854 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36158
I20260129 03:01:08.535676 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260129 03:01:08.538918 27993 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.539690 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260129 03:01:08.539937 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-1-root
uuid: "20f7ff56a6a5447899f5d9025961eaf2"
format_stamp: "Formatted at 2026-01-29 03:01:08 on dist-test-slave-1zcf"
I20260129 03:01:08.540170 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:01:08.552386 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:01:08.553380 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:01:08.554777 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:01:08.556923 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:01:08.557101 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.557317 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:01:08.557452 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.594444 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.66:37651
I20260129 03:01:08.594540 28055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.66:37651 every 8 connection(s)
I20260129 03:01:08.598915 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:01:08.605486 28059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:08.606739 28060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:08.608485 28062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:08.608856 23477 server_base.cc:1060] running on GCE node
I20260129 03:01:08.610013 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:01:08.610268 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:01:08.610374 28056 heartbeater.cc:344] Connected to a master server at 127.22.237.126:45981
I20260129 03:01:08.610431 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655668610416 us; error 0 us; skew 500 ppm
I20260129 03:01:08.610826 28056 heartbeater.cc:461] Registering TS with master...
I20260129 03:01:08.611198 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:01:08.611545 28056 heartbeater.cc:507] Master 127.22.237.126:45981 requested a full tablet report, sending...
I20260129 03:01:08.613451 27854 ts_manager.cc:194] Registered new tserver with Master: 20f7ff56a6a5447899f5d9025961eaf2 (127.22.237.66:37651)
I20260129 03:01:08.613905 23477 webserver.cc:492] Webserver started at http://127.22.237.67:34103/ using document root <none> and password file <none>
I20260129 03:01:08.614382 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:01:08.614554 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:01:08.614790 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:01:08.614926 27854 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36160
I20260129 03:01:08.615808 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-2-root/instance:
uuid: "f4e4133cd4534353a276277e0e2fab31"
format_stamp: "Formatted at 2026-01-29 03:01:08 on dist-test-slave-1zcf"
I20260129 03:01:08.619994 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260129 03:01:08.623180 28067 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.623924 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260129 03:01:08.624199 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-2-root
uuid: "f4e4133cd4534353a276277e0e2fab31"
format_stamp: "Formatted at 2026-01-29 03:01:08 on dist-test-slave-1zcf"
I20260129 03:01:08.624457 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:01:08.648738 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:01:08.649821 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:01:08.651238 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:01:08.653358 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:01:08.653553 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.653789 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:01:08.653944 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.690232 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.67:34465
I20260129 03:01:08.690395 28129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.67:34465 every 8 connection(s)
I20260129 03:01:08.694629 23477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260129 03:01:08.701021 28133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260129 03:01:08.702844 28134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:08.705971 28130 heartbeater.cc:344] Connected to a master server at 127.22.237.126:45981
I20260129 03:01:08.706480 28130 heartbeater.cc:461] Registering TS with master...
I20260129 03:01:08.707494 28130 heartbeater.cc:507] Master 127.22.237.126:45981 requested a full tablet report, sending...
I20260129 03:01:08.709969 27854 ts_manager.cc:194] Registered new tserver with Master: f4e4133cd4534353a276277e0e2fab31 (127.22.237.67:34465)
I20260129 03:01:08.710348 23477 server_base.cc:1060] running on GCE node
W20260129 03:01:08.711169 28137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260129 03:01:08.711585 27854 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36174
I20260129 03:01:08.712168 23477 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260129 03:01:08.712445 23477 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260129 03:01:08.712638 23477 hybrid_clock.cc:648] HybridClock initialized: now 1769655668712620 us; error 0 us; skew 500 ppm
I20260129 03:01:08.713343 23477 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260129 03:01:08.715626 23477 webserver.cc:492] Webserver started at http://127.22.237.68:40347/ using document root <none> and password file <none>
I20260129 03:01:08.716058 23477 fs_manager.cc:362] Metadata directory not provided
I20260129 03:01:08.716208 23477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260129 03:01:08.716413 23477 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260129 03:01:08.717381 23477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-3-root/instance:
uuid: "acd111d368504bca817b2ac21eb7970e"
format_stamp: "Formatted at 2026-01-29 03:01:08 on dist-test-slave-1zcf"
I20260129 03:01:08.721510 23477 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260129 03:01:08.724825 28141 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.725589 23477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260129 03:01:08.725970 23477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-3-root
uuid: "acd111d368504bca817b2ac21eb7970e"
format_stamp: "Formatted at 2026-01-29 03:01:08 on dist-test-slave-1zcf"
I20260129 03:01:08.726256 23477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260129 03:01:08.737814 23477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260129 03:01:08.738863 23477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260129 03:01:08.740236 23477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260129 03:01:08.742482 23477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260129 03:01:08.742679 23477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.742916 23477 ts_tablet_manager.cc:616] Registered 0 tablets
I20260129 03:01:08.743059 23477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260129 03:01:08.780316 23477 rpc_server.cc:307] RPC server started. Bound to: 127.22.237.68:35051
I20260129 03:01:08.780402 28203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.237.68:35051 every 8 connection(s)
I20260129 03:01:08.793051 28204 heartbeater.cc:344] Connected to a master server at 127.22.237.126:45981
I20260129 03:01:08.793403 28204 heartbeater.cc:461] Registering TS with master...
I20260129 03:01:08.794144 28204 heartbeater.cc:507] Master 127.22.237.126:45981 requested a full tablet report, sending...
I20260129 03:01:08.795871 27854 ts_manager.cc:194] Registered new tserver with Master: acd111d368504bca817b2ac21eb7970e (127.22.237.68:35051)
I20260129 03:01:08.796036 23477 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.012458675s
I20260129 03:01:08.797240 27854 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36188
I20260129 03:01:08.836289 27854 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36206:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260129 03:01:08.874136 28095 tablet_service.cc:1505] Processing CreateTablet for tablet f11ee25c4a7c4e0a84a2de6b92556ff5 (DEFAULT_TABLE table=test-workload [id=e5d836c4496c47c5b3e7c663918b4940]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 03:01:08.875413 28095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f11ee25c4a7c4e0a84a2de6b92556ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:08.878151 27946 tablet_service.cc:1505] Processing CreateTablet for tablet f11ee25c4a7c4e0a84a2de6b92556ff5 (DEFAULT_TABLE table=test-workload [id=e5d836c4496c47c5b3e7c663918b4940]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 03:01:08.878322 28169 tablet_service.cc:1505] Processing CreateTablet for tablet f11ee25c4a7c4e0a84a2de6b92556ff5 (DEFAULT_TABLE table=test-workload [id=e5d836c4496c47c5b3e7c663918b4940]), partition=RANGE (key) PARTITION UNBOUNDED
I20260129 03:01:08.879181 27946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f11ee25c4a7c4e0a84a2de6b92556ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:08.879536 28169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f11ee25c4a7c4e0a84a2de6b92556ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:08.891750 28224 tablet_bootstrap.cc:492] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31: Bootstrap starting.
I20260129 03:01:08.894150 28225 tablet_bootstrap.cc:492] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: Bootstrap starting.
I20260129 03:01:08.896201 28226 tablet_bootstrap.cc:492] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e: Bootstrap starting.
I20260129 03:01:08.898128 28224 tablet_bootstrap.cc:654] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31: Neither blocks nor log segments found. Creating new log.
I20260129 03:01:08.900231 28225 tablet_bootstrap.cc:654] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: Neither blocks nor log segments found. Creating new log.
I20260129 03:01:08.903437 28224 tablet_bootstrap.cc:492] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31: No bootstrap required, opened a new log
I20260129 03:01:08.903887 28226 tablet_bootstrap.cc:654] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e: Neither blocks nor log segments found. Creating new log.
I20260129 03:01:08.903942 28224 ts_tablet_manager.cc:1403] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260129 03:01:08.904745 28225 tablet_bootstrap.cc:492] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: No bootstrap required, opened a new log
I20260129 03:01:08.905115 28225 ts_tablet_manager.cc:1403] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260129 03:01:08.906996 28224 raft_consensus.cc:359] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } }
I20260129 03:01:08.907263 28225 raft_consensus.cc:359] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } }
I20260129 03:01:08.907812 28224 raft_consensus.cc:385] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:01:08.908021 28225 raft_consensus.cc:385] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:01:08.908218 28224 raft_consensus.cc:740] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e4133cd4534353a276277e0e2fab31, State: Initialized, Role: FOLLOWER
I20260129 03:01:08.908411 28225 raft_consensus.cc:740] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7fdf8fe04f44d0b1e8e8c9d82d3c08, State: Initialized, Role: FOLLOWER
I20260129 03:01:08.909101 28224 consensus_queue.cc:260] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } }
I20260129 03:01:08.909299 28225 consensus_queue.cc:260] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } }
I20260129 03:01:08.911288 28130 heartbeater.cc:499] Master 127.22.237.126:45981 was elected leader, sending a full tablet report...
I20260129 03:01:08.911360 28226 tablet_bootstrap.cc:492] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e: No bootstrap required, opened a new log
I20260129 03:01:08.911875 28224 ts_tablet_manager.cc:1434] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260129 03:01:08.912170 28226 ts_tablet_manager.cc:1403] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260129 03:01:08.913620 27981 heartbeater.cc:499] Master 127.22.237.126:45981 was elected leader, sending a full tablet report...
I20260129 03:01:08.914276 28225 ts_tablet_manager.cc:1434] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260129 03:01:08.915333 28226 raft_consensus.cc:359] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } }
I20260129 03:01:08.915995 28226 raft_consensus.cc:385] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260129 03:01:08.916220 28226 raft_consensus.cc:740] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd111d368504bca817b2ac21eb7970e, State: Initialized, Role: FOLLOWER
I20260129 03:01:08.916821 28226 consensus_queue.cc:260] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } }
I20260129 03:01:08.919394 28226 ts_tablet_manager.cc:1434] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260129 03:01:08.919775 28204 heartbeater.cc:499] Master 127.22.237.126:45981 was elected leader, sending a full tablet report...
I20260129 03:01:09.097837 28231 raft_consensus.cc:493] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260129 03:01:09.098378 28231 raft_consensus.cc:515] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } }
I20260129 03:01:09.100306 28231 leader_election.cc:290] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4e4133cd4534353a276277e0e2fab31 (127.22.237.67:34465), acd111d368504bca817b2ac21eb7970e (127.22.237.68:35051)
I20260129 03:01:09.109215 28105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f11ee25c4a7c4e0a84a2de6b92556ff5" candidate_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4e4133cd4534353a276277e0e2fab31" is_pre_election: true
I20260129 03:01:09.110041 28105 raft_consensus.cc:2468] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 in term 0.
I20260129 03:01:09.110100 28179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f11ee25c4a7c4e0a84a2de6b92556ff5" candidate_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd111d368504bca817b2ac21eb7970e" is_pre_election: true
I20260129 03:01:09.110682 28179 raft_consensus.cc:2468] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 in term 0.
I20260129 03:01:09.110957 27919 leader_election.cc:304] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7fdf8fe04f44d0b1e8e8c9d82d3c08, f4e4133cd4534353a276277e0e2fab31; no voters:
I20260129 03:01:09.111683 28231 raft_consensus.cc:2804] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260129 03:01:09.112058 28231 raft_consensus.cc:493] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260129 03:01:09.112316 28231 raft_consensus.cc:3060] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:01:09.117377 28231 raft_consensus.cc:515] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } }
I20260129 03:01:09.118906 28231 leader_election.cc:290] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [CANDIDATE]: Term 1 election: Requested vote from peers f4e4133cd4534353a276277e0e2fab31 (127.22.237.67:34465), acd111d368504bca817b2ac21eb7970e (127.22.237.68:35051)
I20260129 03:01:09.119670 28105 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f11ee25c4a7c4e0a84a2de6b92556ff5" candidate_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4e4133cd4534353a276277e0e2fab31"
I20260129 03:01:09.119899 28179 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f11ee25c4a7c4e0a84a2de6b92556ff5" candidate_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd111d368504bca817b2ac21eb7970e"
I20260129 03:01:09.120277 28105 raft_consensus.cc:3060] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:01:09.120399 28179 raft_consensus.cc:3060] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e [term 0 FOLLOWER]: Advancing to term 1
I20260129 03:01:09.127245 28105 raft_consensus.cc:2468] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 in term 1.
I20260129 03:01:09.127245 28179 raft_consensus.cc:2468] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 in term 1.
I20260129 03:01:09.128523 27919 leader_election.cc:304] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7fdf8fe04f44d0b1e8e8c9d82d3c08, f4e4133cd4534353a276277e0e2fab31; no voters:
I20260129 03:01:09.129174 28231 raft_consensus.cc:2804] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 1 FOLLOWER]: Leader election won for term 1
I20260129 03:01:09.130105 28231 raft_consensus.cc:697] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 1 LEADER]: Becoming Leader. State: Replica: 2a7fdf8fe04f44d0b1e8e8c9d82d3c08, State: Running, Role: LEADER
I20260129 03:01:09.130833 28231 consensus_queue.cc:237] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } }
I20260129 03:01:09.137709 27853 catalog_manager.cc:5654] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 (127.22.237.65). New cstate: current_term: 1 leader_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } health_report { overall_health: HEALTHY } } }
I20260129 03:01:09.406633 27946 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260129 03:01:09.407187 28169 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:09.409708 28021 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:09.409780 28095 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:09.617250 28056 heartbeater.cc:499] Master 127.22.237.126:45981 was elected leader, sending a full tablet report...
I20260129 03:01:09.662824 28267 consensus_queue.cc:1048] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd111d368504bca817b2ac21eb7970e" member_type: VOTER last_known_addr { host: "127.22.237.68" port: 35051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3957c37ca90e4228a0b77e60c9f742df | 127.22.237.126:45981 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
20f7ff56a6a5447899f5d9025961eaf2 | 127.22.237.66:37651 | HEALTHY | <none> | 0 | 0
2a7fdf8fe04f44d0b1e8e8c9d82d3c08 | 127.22.237.65:34501 | HEALTHY | <none> | 1 | 0
acd111d368504bca817b2ac21eb7970e | 127.22.237.68:35051 | HEALTHY | <none> | 0 | 0
f4e4133cd4534353a276277e0e2fab31 | 127.22.237.67:34465 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet f11ee25c4a7c4e0a84a2de6b92556ff5 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
f4e4133cd4534353a276277e0e2fab31 (127.22.237.67:34465): RUNNING
acd111d368504bca817b2ac21eb7970e (127.22.237.68:35051): RUNNING
2a7fdf8fe04f44d0b1e8e8c9d82d3c08 (127.22.237.65:34501): RUNNING [LEADER]
All reported replicas are:
A = f4e4133cd4534353a276277e0e2fab31
B = acd111d368504bca817b2ac21eb7970e
C = 2a7fdf8fe04f44d0b1e8e8c9d82d3c08
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 20f7ff56a6a5447899f5d9025961eaf2 | 127.22.237.66:37651 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260129 03:01:09.683998 23477 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260129 03:01:09.690014 28267 consensus_queue.cc:1048] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260129 03:01:10.060909 27946 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260129 03:01:10.062038 28095 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:10.062062 28021 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:10.064476 28169 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3957c37ca90e4228a0b77e60c9f742df | 127.22.237.126:45981 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
20f7ff56a6a5447899f5d9025961eaf2 | 127.22.237.66:37651 | HEALTHY | <none> | 0 | 0
2a7fdf8fe04f44d0b1e8e8c9d82d3c08 | 127.22.237.65:34501 | HEALTHY | <none> | 1 | 0
acd111d368504bca817b2ac21eb7970e | 127.22.237.68:35051 | HEALTHY | <none> | 0 | 0
f4e4133cd4534353a276277e0e2fab31 | 127.22.237.67:34465 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 20f7ff56a6a5447899f5d9025961eaf2 | 127.22.237.66:37651 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260129 03:01:10.393532 23477 log_verifier.cc:126] Checking tablet f11ee25c4a7c4e0a84a2de6b92556ff5
I20260129 03:01:10.450857 23477 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f11ee25c4a7c4e0a84a2de6b92556ff5
I20260129 03:01:10.451362 23477 tablet_replica.cc:333] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e: stopping tablet replica
I20260129 03:01:10.451917 23477 raft_consensus.cc:2243] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 03:01:10.452454 23477 raft_consensus.cc:2272] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:10.676930 28021 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:10.687549 28169 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:10.687587 27946 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260129 03:01:10.694514 27921 consensus_queue.cc:579] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [LEADER]: The tablet replica hosted on peer acd111d368504bca817b2ac21eb7970e has failed
I20260129 03:01:10.695214 28095 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
W20260129 03:01:10.695165 27921 consensus_peers.cc:597] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 -> Peer acd111d368504bca817b2ac21eb7970e (127.22.237.68:35051): Couldn't send request to peer acd111d368504bca817b2ac21eb7970e. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260129 03:01:10.704352 27956 consensus_queue.cc:237] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } }
I20260129 03:01:10.712024 28105 raft_consensus.cc:1275] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 1 FOLLOWER]: Refusing update from remote peer 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260129 03:01:10.713505 28309 consensus_queue.cc:1048] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260129 03:01:10.720515 28308 raft_consensus.cc:2955] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER acd111d368504bca817b2ac21eb7970e (127.22.237.68) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } } }
I20260129 03:01:10.727412 28105 raft_consensus.cc:2955] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER acd111d368504bca817b2ac21eb7970e (127.22.237.68) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } } }
I20260129 03:01:10.741035 27839 catalog_manager.cc:5167] ChangeConfig:REMOVE_PEER RPC for tablet f11ee25c4a7c4e0a84a2de6b92556ff5 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260129 03:01:10.746760 27853 catalog_manager.cc:5654] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 reported cstate change: config changed from index -1 to 2, VOTER acd111d368504bca817b2ac21eb7970e (127.22.237.68) evicted. New cstate: current_term: 1 leader_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } health_report { overall_health: HEALTHY } } }
I20260129 03:01:10.834760 28169 tablet_service.cc:1552] Processing DeleteTablet for tablet f11ee25c4a7c4e0a84a2de6b92556ff5 with delete_type TABLET_DATA_TOMBSTONED (TS acd111d368504bca817b2ac21eb7970e not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:33116
I20260129 03:01:10.836409 27956 consensus_queue.cc:237] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } } peers { permanent_uuid: "20f7ff56a6a5447899f5d9025961eaf2" member_type: NON_VOTER last_known_addr { host: "127.22.237.66" port: 37651 } attrs { promote: true } }
I20260129 03:01:10.857895 28105 raft_consensus.cc:1275] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 1 FOLLOWER]: Refusing update from remote peer 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260129 03:01:10.859597 28308 consensus_queue.cc:1048] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260129 03:01:10.867347 28267 raft_consensus.cc:2955] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 20f7ff56a6a5447899f5d9025961eaf2 (127.22.237.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } } peers { permanent_uuid: "20f7ff56a6a5447899f5d9025961eaf2" member_type: NON_VOTER last_known_addr { host: "127.22.237.66" port: 37651 } attrs { promote: true } } }
I20260129 03:01:10.876127 28105 raft_consensus.cc:2955] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 20f7ff56a6a5447899f5d9025961eaf2 (127.22.237.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } } peers { permanent_uuid: "20f7ff56a6a5447899f5d9025961eaf2" member_type: NON_VOTER last_known_addr { host: "127.22.237.66" port: 37651 } attrs { promote: true } } }
I20260129 03:01:10.895543 27839 catalog_manager.cc:5167] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f11ee25c4a7c4e0a84a2de6b92556ff5 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260129 03:01:10.901212 27853 catalog_manager.cc:5654] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 reported cstate change: config changed from index 2 to 3, NON_VOTER 20f7ff56a6a5447899f5d9025961eaf2 (127.22.237.66) added. New cstate: current_term: 1 leader_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20f7ff56a6a5447899f5d9025961eaf2" member_type: NON_VOTER last_known_addr { host: "127.22.237.66" port: 37651 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260129 03:01:10.920137 28345 ts_tablet_manager.cc:1916] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260129 03:01:10.933389 27921 consensus_peers.cc:597] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 -> Peer 20f7ff56a6a5447899f5d9025961eaf2 (127.22.237.66:37651): Couldn't send request to peer 20f7ff56a6a5447899f5d9025961eaf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f11ee25c4a7c4e0a84a2de6b92556ff5. This is attempt 1: this message will repeat every 5th retry.
I20260129 03:01:10.947580 28345 ts_tablet_manager.cc:1929] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260129 03:01:10.948124 28345 log.cc:1199] T f11ee25c4a7c4e0a84a2de6b92556ff5 P acd111d368504bca817b2ac21eb7970e: Deleting WAL directory at /tmp/dist-test-taskAigTTw/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1769655599110862-23477-0/minicluster-data/ts-3-root/wals/f11ee25c4a7c4e0a84a2de6b92556ff5
I20260129 03:01:10.949980 27840 catalog_manager.cc:4985] TS acd111d368504bca817b2ac21eb7970e (127.22.237.68:35051): tablet f11ee25c4a7c4e0a84a2de6b92556ff5 (table test-workload [id=e5d836c4496c47c5b3e7c663918b4940]) successfully deleted
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3957c37ca90e4228a0b77e60c9f742df | 127.22.237.126:45981 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
20f7ff56a6a5447899f5d9025961eaf2 | 127.22.237.66:37651 | HEALTHY | <none> | 0 | 0
2a7fdf8fe04f44d0b1e8e8c9d82d3c08 | 127.22.237.65:34501 | HEALTHY | <none> | 1 | 0
acd111d368504bca817b2ac21eb7970e | 127.22.237.68:35051 | HEALTHY | <none> | 0 | 0
f4e4133cd4534353a276277e0e2fab31 | 127.22.237.67:34465 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet f11ee25c4a7c4e0a84a2de6b92556ff5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
f4e4133cd4534353a276277e0e2fab31 (127.22.237.67:34465): RUNNING
acd111d368504bca817b2ac21eb7970e (127.22.237.68:35051): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
2a7fdf8fe04f44d0b1e8e8c9d82d3c08 (127.22.237.65:34501): RUNNING [LEADER]
All reported replicas are:
A = f4e4133cd4534353a276277e0e2fab31
B = acd111d368504bca817b2ac21eb7970e
C = 2a7fdf8fe04f44d0b1e8e8c9d82d3c08
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | [config not available] | | |
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 20f7ff56a6a5447899f5d9025961eaf2 | 127.22.237.66:37651 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260129 03:01:11.033915 23477 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260129 03:01:11.390678 28021 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:11.401916 28095 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:11.404878 28375 ts_tablet_manager.cc:933] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: Initiating tablet copy from peer 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 (127.22.237.65:34501)
I20260129 03:01:11.405488 28169 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260129 03:01:11.406522 27946 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260129 03:01:11.466033 28375 tablet_copy_client.cc:323] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: tablet copy: Beginning tablet copy session from remote peer at address 127.22.237.65:34501
I20260129 03:01:11.489355 27966 tablet_copy_service.cc:140] P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: Received BeginTabletCopySession request for tablet f11ee25c4a7c4e0a84a2de6b92556ff5 from peer 20f7ff56a6a5447899f5d9025961eaf2 ({username='slave'} at 127.0.0.1:54646)
I20260129 03:01:11.490017 27966 tablet_copy_service.cc:161] P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: Beginning new tablet copy session on tablet f11ee25c4a7c4e0a84a2de6b92556ff5 from peer 20f7ff56a6a5447899f5d9025961eaf2 at {username='slave'} at 127.0.0.1:54646: session id = 20f7ff56a6a5447899f5d9025961eaf2-f11ee25c4a7c4e0a84a2de6b92556ff5
I20260129 03:01:11.497946 27966 tablet_copy_source_session.cc:215] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: Tablet Copy: opened 0 blocks and 1 log segments
I20260129 03:01:11.506323 28375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f11ee25c4a7c4e0a84a2de6b92556ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260129 03:01:11.549052 28375 tablet_copy_client.cc:806] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: tablet copy: Starting download of 0 data blocks...
I20260129 03:01:11.549798 28375 tablet_copy_client.cc:670] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: tablet copy: Starting download of 1 WAL segments...
I20260129 03:01:11.557374 28375 tablet_copy_client.cc:538] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260129 03:01:11.566694 28375 tablet_bootstrap.cc:492] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: Bootstrap starting.
I20260129 03:01:11.608222 28375 tablet_bootstrap.cc:492] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260129 03:01:11.609112 28375 tablet_bootstrap.cc:492] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: Bootstrap complete.
I20260129 03:01:11.609856 28375 ts_tablet_manager.cc:1403] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: Time spent bootstrapping tablet: real 0.043s user 0.022s sys 0.011s
I20260129 03:01:11.612778 28375 raft_consensus.cc:359] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } } peers { permanent_uuid: "20f7ff56a6a5447899f5d9025961eaf2" member_type: NON_VOTER last_known_addr { host: "127.22.237.66" port: 37651 } attrs { promote: true } }
I20260129 03:01:11.613534 28375 raft_consensus.cc:740] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20f7ff56a6a5447899f5d9025961eaf2, State: Initialized, Role: LEARNER
I20260129 03:01:11.614248 28375 consensus_queue.cc:260] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f4e4133cd4534353a276277e0e2fab31" member_type: VOTER last_known_addr { host: "127.22.237.67" port: 34465 } } peers { permanent_uuid: "2a7fdf8fe04f44d0b1e8e8c9d82d3c08" member_type: VOTER last_known_addr { host: "127.22.237.65" port: 34501 } } peers { permanent_uuid: "20f7ff56a6a5447899f5d9025961eaf2" member_type: NON_VOTER last_known_addr { host: "127.22.237.66" port: 37651 } attrs { promote: true } }
I20260129 03:01:11.617009 28375 ts_tablet_manager.cc:1434] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260129 03:01:11.618728 27966 tablet_copy_service.cc:342] P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: Request end of tablet copy session 20f7ff56a6a5447899f5d9025961eaf2-f11ee25c4a7c4e0a84a2de6b92556ff5 received from {username='slave'} at 127.0.0.1:54646
I20260129 03:01:11.619143 27966 tablet_copy_service.cc:434] P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: ending tablet copy session 20f7ff56a6a5447899f5d9025961eaf2-f11ee25c4a7c4e0a84a2de6b92556ff5 on tablet f11ee25c4a7c4e0a84a2de6b92556ff5 with peer 20f7ff56a6a5447899f5d9025961eaf2
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3957c37ca90e4228a0b77e60c9f742df | 127.22.237.126:45981 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
20f7ff56a6a5447899f5d9025961eaf2 | 127.22.237.66:37651 | HEALTHY | <none> | 0 | 0
2a7fdf8fe04f44d0b1e8e8c9d82d3c08 | 127.22.237.65:34501 | HEALTHY | <none> | 1 | 0
acd111d368504bca817b2ac21eb7970e | 127.22.237.68:35051 | HEALTHY | <none> | 0 | 0
f4e4133cd4534353a276277e0e2fab31 | 127.22.237.67:34465 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | acd111d368504bca817b2ac21eb7970e | 127.22.237.68:35051 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260129 03:01:11.673707 23477 log_verifier.cc:126] Checking tablet f11ee25c4a7c4e0a84a2de6b92556ff5
I20260129 03:01:11.733919 23477 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f11ee25c4a7c4e0a84a2de6b92556ff5
I20260129 03:01:11.737550 23477 tablet_server.cc:178] TabletServer@127.22.237.65:0 shutting down...
I20260129 03:01:11.759756 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:01:11.760505 23477 tablet_replica.cc:333] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08: stopping tablet replica
I20260129 03:01:11.761066 23477 raft_consensus.cc:2243] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:01:11.761904 23477 raft_consensus.cc:2272] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 2a7fdf8fe04f44d0b1e8e8c9d82d3c08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:11.782281 23477 tablet_server.cc:195] TabletServer@127.22.237.65:0 shutdown complete.
I20260129 03:01:11.793305 23477 tablet_server.cc:178] TabletServer@127.22.237.66:0 shutting down...
I20260129 03:01:11.811574 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:01:11.812438 23477 tablet_replica.cc:333] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2: stopping tablet replica
I20260129 03:01:11.813104 23477 raft_consensus.cc:2243] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2 [term 1 LEARNER]: Raft consensus shutting down.
I20260129 03:01:11.813570 23477 raft_consensus.cc:2272] T f11ee25c4a7c4e0a84a2de6b92556ff5 P 20f7ff56a6a5447899f5d9025961eaf2 [term 1 LEARNER]: Raft consensus is shut down!
I20260129 03:01:11.832969 23477 tablet_server.cc:195] TabletServer@127.22.237.66:0 shutdown complete.
I20260129 03:01:11.842650 23477 tablet_server.cc:178] TabletServer@127.22.237.67:0 shutting down...
I20260129 03:01:11.861492 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:01:11.862205 23477 tablet_replica.cc:333] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31: stopping tablet replica
I20260129 03:01:11.862692 23477 raft_consensus.cc:2243] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260129 03:01:11.863204 23477 raft_consensus.cc:2272] T f11ee25c4a7c4e0a84a2de6b92556ff5 P f4e4133cd4534353a276277e0e2fab31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:11.882954 23477 tablet_server.cc:195] TabletServer@127.22.237.67:0 shutdown complete.
I20260129 03:01:11.892817 23477 tablet_server.cc:178] TabletServer@127.22.237.68:0 shutting down...
I20260129 03:01:11.910344 23477 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260129 03:01:11.926997 23477 tablet_server.cc:195] TabletServer@127.22.237.68:0 shutdown complete.
I20260129 03:01:11.936650 23477 master.cc:561] Master@127.22.237.126:45981 shutting down...
I20260129 03:01:11.952499 23477 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 1 LEADER]: Raft consensus shutting down.
I20260129 03:01:11.953027 23477 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260129 03:01:11.953336 23477 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3957c37ca90e4228a0b77e60c9f742df: stopping tablet replica
I20260129 03:01:11.970754 23477 master.cc:583] Master@127.22.237.126:45981 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (3759 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (11701 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (72726 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20260129 03:01:12.006762 23477 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~15 seconds ago
I20260129 03:01:12.007045 23477 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 115 messages since previous log ~64 seconds ago
I20260129 03:01:12.007218 23477 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 6 messages since previous log ~19 seconds ago
I20260129 03:01:12.007495 23477 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 8 messages since previous log ~18 seconds ago