[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260318 08:59:16.813551 6153 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.174s user 0.410s sys 0.762s
W20260318 08:59:16.814038 6153 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.175s user 0.410s sys 0.762s
I20260318 08:59:16.817044 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:46835
I20260318 08:59:16.818807 6153 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260318 08:59:16.820108 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:16.832616 6164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:16.834302 6165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:16.986057 6167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:16.986063 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:16.987418 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:16.987658 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:16.987861 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824356987838 us; error 0 us; skew 500 ppm
I20260318 08:59:16.988685 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:16.993284 6153 webserver.cc:492] Webserver started at http://127.6.2.126:33735/ using document root <none> and password file <none>
I20260318 08:59:16.994031 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:16.994251 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:16.994668 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:17.000201 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "356d9450997d49a6a3bac4b047d90045"
format_stamp: "Formatted at 2026-03-18 08:59:16 on dist-test-slave-mpvv"
I20260318 08:59:17.007042 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260318 08:59:17.011832 6172 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:17.012856 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260318 08:59:17.013185 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "356d9450997d49a6a3bac4b047d90045"
format_stamp: "Formatted at 2026-03-18 08:59:16 on dist-test-slave-mpvv"
I20260318 08:59:17.013515 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:17.044662 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:17.046161 6153 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260318 08:59:17.046583 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:17.114262 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:46835
I20260318 08:59:17.114333 6223 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:46835 every 8 connection(s)
I20260318 08:59:17.120702 6224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:17.141505 6224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045: Bootstrap starting.
I20260318 08:59:17.147691 6224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:17.149459 6224 log.cc:826] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045: Log is configured to *not* fsync() on all Append() calls
I20260318 08:59:17.153729 6224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045: No bootstrap required, opened a new log
I20260318 08:59:17.173226 6224 raft_consensus.cc:359] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356d9450997d49a6a3bac4b047d90045" member_type: VOTER }
I20260318 08:59:17.173753 6224 raft_consensus.cc:385] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:17.174058 6224 raft_consensus.cc:740] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356d9450997d49a6a3bac4b047d90045, State: Initialized, Role: FOLLOWER
I20260318 08:59:17.174842 6224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356d9450997d49a6a3bac4b047d90045" member_type: VOTER }
I20260318 08:59:17.175345 6224 raft_consensus.cc:399] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 08:59:17.175652 6224 raft_consensus.cc:493] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 08:59:17.176002 6224 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:17.180287 6224 raft_consensus.cc:515] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356d9450997d49a6a3bac4b047d90045" member_type: VOTER }
I20260318 08:59:17.180981 6224 leader_election.cc:304] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 356d9450997d49a6a3bac4b047d90045; no voters:
I20260318 08:59:17.182633 6224 leader_election.cc:290] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 08:59:17.183014 6227 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 08:59:17.185369 6227 raft_consensus.cc:697] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 1 LEADER]: Becoming Leader. State: Replica: 356d9450997d49a6a3bac4b047d90045, State: Running, Role: LEADER
I20260318 08:59:17.186161 6227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356d9450997d49a6a3bac4b047d90045" member_type: VOTER }
I20260318 08:59:17.187103 6224 sys_catalog.cc:565] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [sys.catalog]: configured and running, proceeding with master startup.
I20260318 08:59:17.193207 6229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 356d9450997d49a6a3bac4b047d90045. Latest consensus state: current_term: 1 leader_uuid: "356d9450997d49a6a3bac4b047d90045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356d9450997d49a6a3bac4b047d90045" member_type: VOTER } }
I20260318 08:59:17.193861 6229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:17.193905 6228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "356d9450997d49a6a3bac4b047d90045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356d9450997d49a6a3bac4b047d90045" member_type: VOTER } }
I20260318 08:59:17.194424 6228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:17.198204 6235 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 08:59:17.207952 6235 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 08:59:17.211084 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 08:59:17.223429 6235 catalog_manager.cc:1357] Generated new cluster ID: fa217a5fe01342d18d6ee00dfa341c63
I20260318 08:59:17.223655 6235 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 08:59:17.235298 6235 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 08:59:17.236599 6235 catalog_manager.cc:1514] Loading token signing keys...
I20260318 08:59:17.251745 6235 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045: Generated new TSK 0
I20260318 08:59:17.252524 6235 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 08:59:17.280316 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:17.288934 6246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:17.290217 6247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:17.293716 6249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:17.293771 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:17.294814 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:17.295069 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:17.295295 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824357295279 us; error 0 us; skew 500 ppm
I20260318 08:59:17.296031 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:17.299897 6153 webserver.cc:492] Webserver started at http://127.6.2.65:38919/ using document root <none> and password file <none>
I20260318 08:59:17.300590 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:17.300843 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:17.301229 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:17.302788 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "919f70d75ce349c79e5a3bb762a8b6f9"
format_stamp: "Formatted at 2026-03-18 08:59:17 on dist-test-slave-mpvv"
I20260318 08:59:17.309329 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260318 08:59:17.314138 6254 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:17.314913 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260318 08:59:17.315246 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "919f70d75ce349c79e5a3bb762a8b6f9"
format_stamp: "Formatted at 2026-03-18 08:59:17 on dist-test-slave-mpvv"
I20260318 08:59:17.315591 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:17.338757 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:17.340583 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:17.343303 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:17.348541 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:17.348842 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:17.349152 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:17.349356 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:17.437603 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:39395
I20260318 08:59:17.437649 6316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:39395 every 8 connection(s)
I20260318 08:59:17.445051 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:17.463620 6321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:17.465889 6322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:17.468487 6317 heartbeater.cc:344] Connected to a master server at 127.6.2.126:46835
I20260318 08:59:17.469036 6317 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:17.470356 6317 heartbeater.cc:507] Master 127.6.2.126:46835 requested a full tablet report, sending...
I20260318 08:59:17.474376 6189 ts_manager.cc:194] Registered new tserver with Master: 919f70d75ce349c79e5a3bb762a8b6f9 (127.6.2.65:39395)
W20260318 08:59:17.476099 6324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:17.478149 6189 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34310
I20260318 08:59:17.480679 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:17.481436 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:17.481644 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:17.481805 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824357481785 us; error 0 us; skew 500 ppm
I20260318 08:59:17.482275 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:17.484954 6153 webserver.cc:492] Webserver started at http://127.6.2.66:37637/ using document root <none> and password file <none>
I20260318 08:59:17.485414 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:17.485608 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:17.485858 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:17.487008 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/ts-1-root/instance:
uuid: "5c3d2ad2566d4708ab237abad4595044"
format_stamp: "Formatted at 2026-03-18 08:59:17 on dist-test-slave-mpvv"
I20260318 08:59:17.492092 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260318 08:59:17.495663 6329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:17.496515 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260318 08:59:17.496846 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/ts-1-root
uuid: "5c3d2ad2566d4708ab237abad4595044"
format_stamp: "Formatted at 2026-03-18 08:59:17 on dist-test-slave-mpvv"
I20260318 08:59:17.497092 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1773824355427976-6153-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:17.511083 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:17.512220 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:17.513375 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:17.515966 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:17.516172 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:17.516395 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:17.516613 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:17.581911 6317 heartbeater.cc:499] Master 127.6.2.126:46835 was elected leader, sending a full tablet report...
I20260318 08:59:17.608757 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.66:40389
I20260318 08:59:17.608889 6391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.66:40389 every 8 connection(s)
I20260318 08:59:17.624857 6392 heartbeater.cc:344] Connected to a master server at 127.6.2.126:46835
I20260318 08:59:17.625227 6392 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:17.626075 6392 heartbeater.cc:507] Master 127.6.2.126:46835 requested a full tablet report, sending...
I20260318 08:59:17.627874 6189 ts_manager.cc:194] Registered new tserver with Master: 5c3d2ad2566d4708ab237abad4595044 (127.6.2.66:40389)
I20260318 08:59:17.628849 6153 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.015527126s
I20260318 08:59:17.629317 6189 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34326
I20260318 08:59:17.660092 6189 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20260318 08:59:17.677626 6189 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260318 08:59:17.726286 6357 tablet_service.cc:1511] Processing CreateTablet for tablet 4a856c5a3cf84ef9887f8edb1e79322d (DEFAULT_TABLE table=test-table [id=502fee168c41437893482830ee9acf33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:17.727926 6357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a856c5a3cf84ef9887f8edb1e79322d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:17.731875 6392 heartbeater.cc:499] Master 127.6.2.126:46835 was elected leader, sending a full tablet report...
I20260318 08:59:17.731818 6356 tablet_service.cc:1511] Processing CreateTablet for tablet c39e5b0f883a43c58ebf43d636ba1689 (DEFAULT_TABLE table=test-table [id=502fee168c41437893482830ee9acf33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:17.732625 6282 tablet_service.cc:1511] Processing CreateTablet for tablet 4a856c5a3cf84ef9887f8edb1e79322d (DEFAULT_TABLE table=test-table [id=502fee168c41437893482830ee9acf33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:17.733125 6356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c39e5b0f883a43c58ebf43d636ba1689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:17.733923 6282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a856c5a3cf84ef9887f8edb1e79322d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:17.739952 6281 tablet_service.cc:1511] Processing CreateTablet for tablet c39e5b0f883a43c58ebf43d636ba1689 (DEFAULT_TABLE table=test-table [id=502fee168c41437893482830ee9acf33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:17.741112 6281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c39e5b0f883a43c58ebf43d636ba1689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:17.760618 6406 tablet_bootstrap.cc:492] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044: Bootstrap starting.
I20260318 08:59:17.764943 6405 tablet_bootstrap.cc:492] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9: Bootstrap starting.
I20260318 08:59:17.767993 6406 tablet_bootstrap.cc:654] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:17.771912 6405 tablet_bootstrap.cc:654] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:17.773351 6406 tablet_bootstrap.cc:492] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044: No bootstrap required, opened a new log
I20260318 08:59:17.773705 6406 ts_tablet_manager.cc:1403] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260318 08:59:17.776079 6406 raft_consensus.cc:359] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:17.776759 6406 raft_consensus.cc:740] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3d2ad2566d4708ab237abad4595044, State: Initialized, Role: FOLLOWER
I20260318 08:59:17.777297 6406 consensus_queue.cc:260] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:17.777809 6405 tablet_bootstrap.cc:492] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9: No bootstrap required, opened a new log
I20260318 08:59:17.778213 6405 ts_tablet_manager.cc:1403] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260318 08:59:17.780809 6405 raft_consensus.cc:359] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:17.781338 6405 raft_consensus.cc:740] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919f70d75ce349c79e5a3bb762a8b6f9, State: Initialized, Role: FOLLOWER
I20260318 08:59:17.781419 6406 ts_tablet_manager.cc:1434] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044: Time spent starting tablet: real 0.007s user 0.003s sys 0.005s
I20260318 08:59:17.781930 6405 consensus_queue.cc:260] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:17.782370 6406 tablet_bootstrap.cc:492] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044: Bootstrap starting.
I20260318 08:59:17.789242 6406 tablet_bootstrap.cc:654] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:17.794531 6405 ts_tablet_manager.cc:1434] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260318 08:59:17.800405 6405 tablet_bootstrap.cc:492] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9: Bootstrap starting.
I20260318 08:59:17.803539 6406 tablet_bootstrap.cc:492] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044: No bootstrap required, opened a new log
I20260318 08:59:17.803957 6406 ts_tablet_manager.cc:1403] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.006s
I20260318 08:59:17.806432 6406 raft_consensus.cc:359] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } } peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } }
I20260318 08:59:17.807111 6406 raft_consensus.cc:740] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3d2ad2566d4708ab237abad4595044, State: Initialized, Role: FOLLOWER
I20260318 08:59:17.807653 6406 consensus_queue.cc:260] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } } peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } }
I20260318 08:59:17.809232 6406 ts_tablet_manager.cc:1434] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260318 08:59:17.811808 6405 tablet_bootstrap.cc:654] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:17.817049 6405 tablet_bootstrap.cc:492] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9: No bootstrap required, opened a new log
I20260318 08:59:17.817484 6405 ts_tablet_manager.cc:1403] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260318 08:59:17.820075 6405 raft_consensus.cc:359] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } } peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } }
I20260318 08:59:17.820619 6405 raft_consensus.cc:740] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919f70d75ce349c79e5a3bb762a8b6f9, State: Initialized, Role: FOLLOWER
I20260318 08:59:17.821079 6405 consensus_queue.cc:260] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } } peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } }
I20260318 08:59:17.823685 6405 ts_tablet_manager.cc:1434] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260318 08:59:17.855027 6153 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20260318 08:59:17.855774 6153 raft_consensus.cc:423] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 0 FOLLOWER]: Emulating election...
I20260318 08:59:17.856079 6153 raft_consensus.cc:3060] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:17.861790 6153 raft_consensus.cc:697] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 1 LEADER]: Becoming Leader. State: Replica: 5c3d2ad2566d4708ab237abad4595044, State: Running, Role: LEADER
I20260318 08:59:17.862469 6153 consensus_queue.cc:237] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } } peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } }
W20260318 08:59:17.864081 6393 tablet.cc:2379] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260318 08:59:17.867803 6153 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260318 08:59:17.874673 6189 catalog_manager.cc:5671] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c3d2ad2566d4708ab237abad4595044 (127.6.2.66). New cstate: current_term: 1 leader_uuid: "5c3d2ad2566d4708ab237abad4595044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } health_report { overall_health: HEALTHY } } }
I20260318 08:59:17.897351 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260318 08:59:18.001245 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260318 08:59:18.204774 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260318 08:59:18.314872 6292 raft_consensus.cc:3060] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:18.321437 6409 consensus_queue.cc:1048] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260318 08:59:18.508308 6153 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260318 08:59:18.508613 6153 raft_consensus.cc:423] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 0 FOLLOWER]: Emulating election...
I20260318 08:59:18.508893 6153 raft_consensus.cc:3060] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:18.514421 6153 raft_consensus.cc:697] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 1 LEADER]: Becoming Leader. State: Replica: 5c3d2ad2566d4708ab237abad4595044, State: Running, Role: LEADER
I20260318 08:59:18.515126 6153 consensus_queue.cc:237] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:18.518980 6153 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260318 08:59:18.520924 6189 catalog_manager.cc:5671] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c3d2ad2566d4708ab237abad4595044 (127.6.2.66). New cstate: current_term: 1 leader_uuid: "5c3d2ad2566d4708ab237abad4595044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } health_report { overall_health: UNKNOWN } } }
I20260318 08:59:18.523918 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260318 08:59:18.627842 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260318 08:59:18.831722 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 0.0 1.1
I20260318 08:59:19.093948 6292 raft_consensus.cc:3060] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:19.100113 6418 consensus_queue.cc:1048] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 08:59:19.336079 6153 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260318 08:59:19.336308 6153 raft_consensus.cc:423] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 1 LEADER]: Emulating election...
I20260318 08:59:19.336615 6153 raft_consensus.cc:3055] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 1 LEADER]: Stepping down as leader of term 1
I20260318 08:59:19.336803 6153 raft_consensus.cc:740] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5c3d2ad2566d4708ab237abad4595044, State: Running, Role: LEADER
I20260318 08:59:19.337380 6153 consensus_queue.cc:260] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:19.338368 6153 raft_consensus.cc:3060] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:19.342355 6153 raft_consensus.cc:697] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 2 LEADER]: Becoming Leader. State: Replica: 5c3d2ad2566d4708ab237abad4595044, State: Running, Role: LEADER
I20260318 08:59:19.342893 6153 consensus_queue.cc:237] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:19.344179 6153 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260318 08:59:19.347448 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260318 08:59:19.347354 6188 catalog_manager.cc:5671] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "5c3d2ad2566d4708ab237abad4595044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } health_report { overall_health: UNKNOWN } } }
I20260318 08:59:19.451269 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260318 08:59:19.655407 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 1.1 2.2
I20260318 08:59:19.829033 6292 raft_consensus.cc:3060] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:19.833835 6292 raft_consensus.cc:1275] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 2 FOLLOWER]: Refusing update from remote peer 5c3d2ad2566d4708ab237abad4595044: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260318 08:59:19.834926 6418 consensus_queue.cc:1048] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260318 08:59:19.958779 6153 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260318 08:59:19.959038 6153 raft_consensus.cc:423] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [term 1 FOLLOWER]: Emulating election...
I20260318 08:59:19.959311 6153 raft_consensus.cc:3060] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:19.962991 6153 raft_consensus.cc:697] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [term 2 LEADER]: Becoming Leader. State: Replica: 919f70d75ce349c79e5a3bb762a8b6f9, State: Running, Role: LEADER
I20260318 08:59:19.963577 6153 consensus_queue.cc:237] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } } peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } }
I20260318 08:59:19.965435 6153 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260318 08:59:19.969094 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260318 08:59:19.969084 6189 catalog_manager.cc:5671] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 reported cstate change: term changed from 1 to 2, leader changed from 5c3d2ad2566d4708ab237abad4595044 (127.6.2.66) to 919f70d75ce349c79e5a3bb762a8b6f9 (127.6.2.65). New cstate: current_term: 2 leader_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } health_report { overall_health: UNKNOWN } } }
I20260318 08:59:20.073235 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260318 08:59:20.268435 6292 raft_consensus.cc:1240] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [term 2 LEADER]: Rejecting Update request from peer 5c3d2ad2566d4708ab237abad4595044 for earlier term 1. Current term is 2. Ops: []
I20260318 08:59:20.269641 6418 consensus_queue.cc:1059] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260318 08:59:20.270427 6418 consensus_queue.cc:1243] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Peer 919f70d75ce349c79e5a3bb762a8b6f9 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260318 08:59:20.270936 6434 raft_consensus.cc:3055] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 1 LEADER]: Stepping down as leader of term 1
I20260318 08:59:20.271291 6434 raft_consensus.cc:740] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5c3d2ad2566d4708ab237abad4595044, State: Running, Role: LEADER
I20260318 08:59:20.271804 6434 consensus_queue.cc:260] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } } peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } }
I20260318 08:59:20.272502 6434 raft_consensus.cc:3060] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:20.279517 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 2.2 1.1
I20260318 08:59:20.408499 6367 raft_consensus.cc:1275] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 2 FOLLOWER]: Refusing update from remote peer 919f70d75ce349c79e5a3bb762a8b6f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260318 08:59:20.409684 6433 consensus_queue.cc:1048] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260318 08:59:20.782920 6153 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20260318 08:59:20.783218 6153 raft_consensus.cc:423] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 2 FOLLOWER]: Emulating election...
I20260318 08:59:20.783480 6153 raft_consensus.cc:3060] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 2 FOLLOWER]: Advancing to term 3
I20260318 08:59:20.787331 6153 raft_consensus.cc:697] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 3 LEADER]: Becoming Leader. State: Replica: 919f70d75ce349c79e5a3bb762a8b6f9, State: Running, Role: LEADER
I20260318 08:59:20.787979 6153 consensus_queue.cc:237] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:20.789330 6153 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260318 08:59:20.792796 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260318 08:59:20.792670 6188 catalog_manager.cc:5671] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 reported cstate change: term changed from 2 to 3, leader changed from 5c3d2ad2566d4708ab237abad4595044 (127.6.2.66) to 919f70d75ce349c79e5a3bb762a8b6f9 (127.6.2.65). New cstate: current_term: 3 leader_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } health_report { overall_health: HEALTHY } } }
I20260318 08:59:20.828140 6292 raft_consensus.cc:1240] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 3 LEADER]: Rejecting Update request from peer 5c3d2ad2566d4708ab237abad4595044 for earlier term 2. Current term is 3. Ops: []
I20260318 08:59:20.829185 6434 consensus_queue.cc:1059] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 08:59:20.830145 6434 consensus_queue.cc:1243] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Peer 919f70d75ce349c79e5a3bb762a8b6f9 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260318 08:59:20.830535 6450 raft_consensus.cc:3055] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 2 LEADER]: Stepping down as leader of term 2
I20260318 08:59:20.830817 6450 raft_consensus.cc:740] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5c3d2ad2566d4708ab237abad4595044, State: Running, Role: LEADER
I20260318 08:59:20.831480 6450 consensus_queue.cc:260] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:20.832235 6450 raft_consensus.cc:3060] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 2 FOLLOWER]: Advancing to term 3
I20260318 08:59:20.896509 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260318 08:59:21.099814 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 2.2
I20260318 08:59:21.324831 6367 raft_consensus.cc:1275] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 3 FOLLOWER]: Refusing update from remote peer 919f70d75ce349c79e5a3bb762a8b6f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260318 08:59:21.325863 6433 consensus_queue.cc:1048] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 08:59:21.403771 6153 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20260318 08:59:21.403995 6153 raft_consensus.cc:423] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 3 FOLLOWER]: Emulating election...
I20260318 08:59:21.404269 6153 raft_consensus.cc:3060] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 3 FOLLOWER]: Advancing to term 4
I20260318 08:59:21.408157 6153 raft_consensus.cc:697] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 4 LEADER]: Becoming Leader. State: Replica: 5c3d2ad2566d4708ab237abad4595044, State: Running, Role: LEADER
I20260318 08:59:21.408774 6153 consensus_queue.cc:237] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:21.410094 6153 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260318 08:59:21.413441 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260318 08:59:21.413197 6189 catalog_manager.cc:5671] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 reported cstate change: term changed from 3 to 4, leader changed from 919f70d75ce349c79e5a3bb762a8b6f9 (127.6.2.65) to 5c3d2ad2566d4708ab237abad4595044 (127.6.2.66). New cstate: current_term: 4 leader_uuid: "5c3d2ad2566d4708ab237abad4595044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } health_report { overall_health: UNKNOWN } } }
I20260318 08:59:21.517087 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260318 08:59:21.720495 6153 cluster_itest_util.cc:258] Not converged past 0 yet: 3.3 4.4
I20260318 08:59:21.814947 6367 raft_consensus.cc:1240] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 4 LEADER]: Rejecting Update request from peer 919f70d75ce349c79e5a3bb762a8b6f9 for earlier term 3. Current term is 4. Ops: []
I20260318 08:59:21.816044 6433 consensus_queue.cc:1059] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260318 08:59:21.816578 6433 consensus_queue.cc:1243] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [LEADER]: Peer 5c3d2ad2566d4708ab237abad4595044 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260318 08:59:21.816912 6452 raft_consensus.cc:3055] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 3 LEADER]: Stepping down as leader of term 3
I20260318 08:59:21.817126 6452 raft_consensus.cc:740] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 919f70d75ce349c79e5a3bb762a8b6f9, State: Running, Role: LEADER
I20260318 08:59:21.817597 6452 consensus_queue.cc:260] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3d2ad2566d4708ab237abad4595044" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 40389 } } peers { permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 } }
I20260318 08:59:21.818426 6452 raft_consensus.cc:3060] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 3 FOLLOWER]: Advancing to term 4
I20260318 08:59:21.959223 6292 raft_consensus.cc:1275] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 4 FOLLOWER]: Refusing update from remote peer 5c3d2ad2566d4708ab237abad4595044: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260318 08:59:21.960602 6458 consensus_queue.cc:1048] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "919f70d75ce349c79e5a3bb762a8b6f9" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260318 08:59:22.225607 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 08:59:22.243528 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:22.244071 6153 tablet_replica.cc:333] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9: stopping tablet replica
I20260318 08:59:22.244607 6153 raft_consensus.cc:2243] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [term 2 LEADER]: Raft consensus shutting down.
I20260318 08:59:22.245137 6153 raft_consensus.cc:2272] T c39e5b0f883a43c58ebf43d636ba1689 P 919f70d75ce349c79e5a3bb762a8b6f9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:22.246876 6153 tablet_replica.cc:333] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9: stopping tablet replica
I20260318 08:59:22.247330 6153 raft_consensus.cc:2243] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260318 08:59:22.247653 6153 raft_consensus.cc:2272] T 4a856c5a3cf84ef9887f8edb1e79322d P 919f70d75ce349c79e5a3bb762a8b6f9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:22.327795 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 08:59:22.336899 6153 tablet_server.cc:178] TabletServer@127.6.2.66:0 shutting down...
I20260318 08:59:22.352782 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:22.353327 6153 tablet_replica.cc:333] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044: stopping tablet replica
I20260318 08:59:22.353817 6153 raft_consensus.cc:2243] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260318 08:59:22.354212 6153 raft_consensus.cc:2272] T c39e5b0f883a43c58ebf43d636ba1689 P 5c3d2ad2566d4708ab237abad4595044 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:22.355588 6153 tablet_replica.cc:333] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044: stopping tablet replica
I20260318 08:59:22.355983 6153 raft_consensus.cc:2243] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 4 LEADER]: Raft consensus shutting down.
I20260318 08:59:22.356483 6153 raft_consensus.cc:2272] T 4a856c5a3cf84ef9887f8edb1e79322d P 5c3d2ad2566d4708ab237abad4595044 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:22.374670 6153 tablet_server.cc:195] TabletServer@127.6.2.66:0 shutdown complete.
I20260318 08:59:22.382555 6153 master.cc:561] Master@127.6.2.126:46835 shutting down...
I20260318 08:59:22.568338 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 1 LEADER]: Raft consensus shutting down.
I20260318 08:59:22.568849 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:22.569167 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 356d9450997d49a6a3bac4b047d90045: stopping tablet replica
I20260318 08:59:22.586184 6153 master.cc:583] Master@127.6.2.126:46835 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (7016 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20260318 08:59:22.610384 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:45337
I20260318 08:59:22.611462 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:22.615566 6466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:22.617113 6467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:22.618252 6153 server_base.cc:1060] running on GCE node
W20260318 08:59:22.618244 6469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:22.619572 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:22.619737 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:22.619845 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824362619835 us; error 0 us; skew 500 ppm
I20260318 08:59:22.620234 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:22.622305 6153 webserver.cc:492] Webserver started at http://127.6.2.126:46273/ using document root <none> and password file <none>
I20260318 08:59:22.622702 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:22.622849 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:22.623036 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:22.623972 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "52bc5262db814e34ae7c35f4f1a2aafd"
format_stamp: "Formatted at 2026-03-18 08:59:22 on dist-test-slave-mpvv"
I20260318 08:59:22.627836 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260318 08:59:22.630774 6474 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:22.631534 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260318 08:59:22.631798 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "52bc5262db814e34ae7c35f4f1a2aafd"
format_stamp: "Formatted at 2026-03-18 08:59:22 on dist-test-slave-mpvv"
I20260318 08:59:22.632046 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:22.645788 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:22.646699 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:22.678480 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:45337
I20260318 08:59:22.678577 6525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:45337 every 8 connection(s)
I20260318 08:59:22.682003 6526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:22.690404 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd: Bootstrap starting.
I20260318 08:59:22.694133 6526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:22.697554 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd: No bootstrap required, opened a new log
I20260318 08:59:22.699255 6526 raft_consensus.cc:359] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc5262db814e34ae7c35f4f1a2aafd" member_type: VOTER }
I20260318 08:59:22.699592 6526 raft_consensus.cc:385] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:22.699764 6526 raft_consensus.cc:740] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bc5262db814e34ae7c35f4f1a2aafd, State: Initialized, Role: FOLLOWER
I20260318 08:59:22.700269 6526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc5262db814e34ae7c35f4f1a2aafd" member_type: VOTER }
I20260318 08:59:22.700663 6526 raft_consensus.cc:399] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 08:59:22.700843 6526 raft_consensus.cc:493] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 08:59:22.701045 6526 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:22.704772 6526 raft_consensus.cc:515] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc5262db814e34ae7c35f4f1a2aafd" member_type: VOTER }
I20260318 08:59:22.705204 6526 leader_election.cc:304] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52bc5262db814e34ae7c35f4f1a2aafd; no voters:
I20260318 08:59:22.706139 6526 leader_election.cc:290] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 08:59:22.706445 6529 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 1 FOLLOWER]: Leader election won for term 1
I20260318 08:59:22.707679 6529 raft_consensus.cc:697] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 1 LEADER]: Becoming Leader. State: Replica: 52bc5262db814e34ae7c35f4f1a2aafd, State: Running, Role: LEADER
I20260318 08:59:22.708338 6529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc5262db814e34ae7c35f4f1a2aafd" member_type: VOTER }
I20260318 08:59:22.708945 6526 sys_catalog.cc:565] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [sys.catalog]: configured and running, proceeding with master startup.
I20260318 08:59:22.711019 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52bc5262db814e34ae7c35f4f1a2aafd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc5262db814e34ae7c35f4f1a2aafd" member_type: VOTER } }
I20260318 08:59:22.711130 6531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52bc5262db814e34ae7c35f4f1a2aafd. Latest consensus state: current_term: 1 leader_uuid: "52bc5262db814e34ae7c35f4f1a2aafd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc5262db814e34ae7c35f4f1a2aafd" member_type: VOTER } }
I20260318 08:59:22.711824 6531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:22.711807 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:22.716276 6535 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 08:59:22.723641 6535 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 08:59:22.724525 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 08:59:22.731627 6535 catalog_manager.cc:1357] Generated new cluster ID: 1245c5e62f6e42eca05014030ecd0fe0
I20260318 08:59:22.731887 6535 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 08:59:22.755260 6535 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 08:59:22.756385 6535 catalog_manager.cc:1514] Loading token signing keys...
I20260318 08:59:22.767915 6535 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd: Generated new TSK 0
I20260318 08:59:22.768460 6535 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 08:59:22.790642 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:22.795519 6547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:22.796800 6548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:22.798107 6550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:22.798349 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:22.799089 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:22.799331 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:22.799466 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824362799455 us; error 0 us; skew 500 ppm
I20260318 08:59:22.799854 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:22.801687 6153 webserver.cc:492] Webserver started at http://127.6.2.65:46613/ using document root <none> and password file <none>
I20260318 08:59:22.802049 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:22.802181 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:22.802366 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:22.803272 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "55f539bce65743919c6e2e899ccc9671"
format_stamp: "Formatted at 2026-03-18 08:59:22 on dist-test-slave-mpvv"
I20260318 08:59:22.806895 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260318 08:59:22.809792 6555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:22.810475 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260318 08:59:22.810758 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "55f539bce65743919c6e2e899ccc9671"
format_stamp: "Formatted at 2026-03-18 08:59:22 on dist-test-slave-mpvv"
I20260318 08:59:22.810994 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:22.824136 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:22.824970 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:22.826164 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:22.828203 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:22.828411 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:22.828640 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:22.828802 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:22.862937 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:43001
I20260318 08:59:22.863021 6617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:43001 every 8 connection(s)
I20260318 08:59:22.867122 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:22.873878 6622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:22.875651 6623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:22.876648 6625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:22.877744 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:22.878473 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:22.878695 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:22.878862 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824362878841 us; error 0 us; skew 500 ppm
I20260318 08:59:22.879415 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:22.882194 6153 webserver.cc:492] Webserver started at http://127.6.2.66:34143/ using document root <none> and password file <none>
I20260318 08:59:22.882814 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:22.883074 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:22.883292 6618 heartbeater.cc:344] Connected to a master server at 127.6.2.126:45337
I20260318 08:59:22.883430 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:22.883659 6618 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:22.884344 6618 heartbeater.cc:507] Master 127.6.2.126:45337 requested a full tablet report, sending...
I20260318 08:59:22.884635 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/ts-1-root/instance:
uuid: "2890048fd67a41c89f37ebbd1f242f53"
format_stamp: "Formatted at 2026-03-18 08:59:22 on dist-test-slave-mpvv"
I20260318 08:59:22.886161 6491 ts_manager.cc:194] Registered new tserver with Master: 55f539bce65743919c6e2e899ccc9671 (127.6.2.65:43001)
I20260318 08:59:22.887776 6491 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41394
I20260318 08:59:22.889341 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260318 08:59:22.892505 6630 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:22.893203 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260318 08:59:22.893474 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/ts-1-root
uuid: "2890048fd67a41c89f37ebbd1f242f53"
format_stamp: "Formatted at 2026-03-18 08:59:22 on dist-test-slave-mpvv"
I20260318 08:59:22.893744 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1773824355427976-6153-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:22.904136 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:22.905035 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:22.906332 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:22.908452 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:22.908702 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:22.908944 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:22.909108 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:22.944792 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.66:44781
I20260318 08:59:22.944877 6692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.66:44781 every 8 connection(s)
I20260318 08:59:22.957191 6693 heartbeater.cc:344] Connected to a master server at 127.6.2.126:45337
I20260318 08:59:22.957540 6693 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:22.958179 6693 heartbeater.cc:507] Master 127.6.2.126:45337 requested a full tablet report, sending...
I20260318 08:59:22.960145 6491 ts_manager.cc:194] Registered new tserver with Master: 2890048fd67a41c89f37ebbd1f242f53 (127.6.2.66:44781)
I20260318 08:59:22.960717 6153 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.012884647s
I20260318 08:59:22.961316 6491 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41402
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:422: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20260318 08:59:22.980037 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 08:59:22.990234 6618 heartbeater.cc:499] Master 127.6.2.126:45337 was elected leader, sending a full tablet report...
I20260318 08:59:22.996948 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:23.011708 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 08:59:23.017752 6153 tablet_server.cc:178] TabletServer@127.6.2.66:0 shutting down...
I20260318 08:59:23.031301 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:23.046195 6153 tablet_server.cc:195] TabletServer@127.6.2.66:0 shutdown complete.
I20260318 08:59:23.052073 6153 master.cc:561] Master@127.6.2.126:45337 shutting down...
I20260318 08:59:23.062860 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 1 LEADER]: Raft consensus shutting down.
I20260318 08:59:23.063330 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:23.063694 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 52bc5262db814e34ae7c35f4f1a2aafd: stopping tablet replica
I20260318 08:59:23.081221 6153 master.cc:583] Master@127.6.2.126:45337 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (491 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20260318 08:59:23.101437 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:37125
I20260318 08:59:23.102336 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:23.106268 6706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:23.107460 6707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:23.109380 6153 server_base.cc:1060] running on GCE node
W20260318 08:59:23.109335 6709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:23.110271 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:23.110488 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:23.110603 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824363110593 us; error 0 us; skew 500 ppm
I20260318 08:59:23.111038 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:23.113080 6153 webserver.cc:492] Webserver started at http://127.6.2.126:40933/ using document root <none> and password file <none>
I20260318 08:59:23.113488 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:23.113631 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:23.113824 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:23.114777 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "7611abd82be64877ae6e5c1bd38798c9"
format_stamp: "Formatted at 2026-03-18 08:59:23 on dist-test-slave-mpvv"
I20260318 08:59:23.118541 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260318 08:59:23.121269 6714 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:23.121884 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260318 08:59:23.122121 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "7611abd82be64877ae6e5c1bd38798c9"
format_stamp: "Formatted at 2026-03-18 08:59:23 on dist-test-slave-mpvv"
I20260318 08:59:23.122373 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:23.146489 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:23.147385 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:23.178997 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:37125
I20260318 08:59:23.179086 6765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:37125 every 8 connection(s)
I20260318 08:59:23.182474 6766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:23.191038 6766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9: Bootstrap starting.
I20260318 08:59:23.194726 6766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:23.198112 6766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9: No bootstrap required, opened a new log
I20260318 08:59:23.199916 6766 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7611abd82be64877ae6e5c1bd38798c9" member_type: VOTER }
I20260318 08:59:23.200266 6766 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:23.200528 6766 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7611abd82be64877ae6e5c1bd38798c9, State: Initialized, Role: FOLLOWER
I20260318 08:59:23.201090 6766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7611abd82be64877ae6e5c1bd38798c9" member_type: VOTER }
I20260318 08:59:23.201519 6766 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 08:59:23.201730 6766 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 08:59:23.201988 6766 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:23.206081 6766 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7611abd82be64877ae6e5c1bd38798c9" member_type: VOTER }
I20260318 08:59:23.206576 6766 leader_election.cc:304] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7611abd82be64877ae6e5c1bd38798c9; no voters:
I20260318 08:59:23.207618 6766 leader_election.cc:290] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 08:59:23.207857 6769 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 08:59:23.209124 6769 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 1 LEADER]: Becoming Leader. State: Replica: 7611abd82be64877ae6e5c1bd38798c9, State: Running, Role: LEADER
I20260318 08:59:23.209770 6769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7611abd82be64877ae6e5c1bd38798c9" member_type: VOTER }
I20260318 08:59:23.210299 6766 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260318 08:59:23.212360 6770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7611abd82be64877ae6e5c1bd38798c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7611abd82be64877ae6e5c1bd38798c9" member_type: VOTER } }
I20260318 08:59:23.212447 6771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7611abd82be64877ae6e5c1bd38798c9. Latest consensus state: current_term: 1 leader_uuid: "7611abd82be64877ae6e5c1bd38798c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7611abd82be64877ae6e5c1bd38798c9" member_type: VOTER } }
I20260318 08:59:23.213014 6771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:23.212994 6770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:23.215332 6775 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 08:59:23.219317 6775 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 08:59:23.227756 6775 catalog_manager.cc:1357] Generated new cluster ID: 34222ce674154b50adb8343fdacf64c8
I20260318 08:59:23.228111 6775 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 08:59:23.229038 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 08:59:23.238840 6775 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 08:59:23.239998 6775 catalog_manager.cc:1514] Loading token signing keys...
I20260318 08:59:23.252684 6775 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9: Generated new TSK 0
I20260318 08:59:23.253227 6775 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 08:59:23.262264 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:23.267330 6787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:23.268321 6788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:23.269641 6790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:23.269903 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:23.270677 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:23.270831 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:23.270938 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824363270928 us; error 0 us; skew 500 ppm
I20260318 08:59:23.271437 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:23.273418 6153 webserver.cc:492] Webserver started at http://127.6.2.65:44605/ using document root <none> and password file <none>
I20260318 08:59:23.273828 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:23.273972 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:23.274160 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:23.275069 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "a68e80718d3b4ef0851babd193edca28"
format_stamp: "Formatted at 2026-03-18 08:59:23 on dist-test-slave-mpvv"
I20260318 08:59:23.279021 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260318 08:59:23.281847 6795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:23.282527 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260318 08:59:23.282783 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "a68e80718d3b4ef0851babd193edca28"
format_stamp: "Formatted at 2026-03-18 08:59:23 on dist-test-slave-mpvv"
I20260318 08:59:23.283041 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:23.304697 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:23.305649 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:23.306941 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:23.308961 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:23.309175 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:23.309407 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:23.309579 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:23.345484 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:35987
I20260318 08:59:23.345547 6857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:35987 every 8 connection(s)
I20260318 08:59:23.349597 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:23.355847 6862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:23.356850 6863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:23.361375 6865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:23.362129 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:23.362250 6858 heartbeater.cc:344] Connected to a master server at 127.6.2.126:37125
I20260318 08:59:23.362684 6858 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:23.363003 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:23.363267 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:23.363334 6858 heartbeater.cc:507] Master 127.6.2.126:37125 requested a full tablet report, sending...
I20260318 08:59:23.363438 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824363363401 us; error 0 us; skew 500 ppm
I20260318 08:59:23.364023 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:23.365417 6731 ts_manager.cc:194] Registered new tserver with Master: a68e80718d3b4ef0851babd193edca28 (127.6.2.65:35987)
I20260318 08:59:23.366605 6153 webserver.cc:492] Webserver started at http://127.6.2.66:41593/ using document root <none> and password file <none>
I20260318 08:59:23.367261 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:23.367499 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:23.367789 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:23.367717 6731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37114
I20260318 08:59:23.369081 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/ts-1-root/instance:
uuid: "a816fb8f841a4f3b82d55d188f475e78"
format_stamp: "Formatted at 2026-03-18 08:59:23 on dist-test-slave-mpvv"
I20260318 08:59:23.373209 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260318 08:59:23.375981 6870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:23.376616 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260318 08:59:23.376881 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/ts-1-root
uuid: "a816fb8f841a4f3b82d55d188f475e78"
format_stamp: "Formatted at 2026-03-18 08:59:23 on dist-test-slave-mpvv"
I20260318 08:59:23.377185 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1773824355427976-6153-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:23.390672 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:23.391650 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:23.392907 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:23.394982 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:23.395217 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:23.395478 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:23.395649 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:23.431310 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.66:38987
I20260318 08:59:23.431365 6932 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.66:38987 every 8 connection(s)
I20260318 08:59:23.443190 6933 heartbeater.cc:344] Connected to a master server at 127.6.2.126:37125
I20260318 08:59:23.443480 6933 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:23.444021 6933 heartbeater.cc:507] Master 127.6.2.126:37125 requested a full tablet report, sending...
I20260318 08:59:23.445631 6731 ts_manager.cc:194] Registered new tserver with Master: a816fb8f841a4f3b82d55d188f475e78 (127.6.2.66:38987)
I20260318 08:59:23.446230 6153 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.011778933s
I20260318 08:59:23.446846 6731 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37124
I20260318 08:59:23.464708 6731 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260318 08:59:23.465981 6731 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260318 08:59:23.470072 6858 heartbeater.cc:499] Master 127.6.2.126:37125 was elected leader, sending a full tablet report...
I20260318 08:59:23.494200 6823 tablet_service.cc:1511] Processing CreateTablet for tablet fabc9dccd785425b91bdba4895c3313b (DEFAULT_TABLE table=test-table [id=e40320077ea34abcbfb65ac8a7e3fec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:23.494848 6898 tablet_service.cc:1511] Processing CreateTablet for tablet fabc9dccd785425b91bdba4895c3313b (DEFAULT_TABLE table=test-table [id=e40320077ea34abcbfb65ac8a7e3fec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:23.495451 6823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fabc9dccd785425b91bdba4895c3313b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:23.495963 6898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fabc9dccd785425b91bdba4895c3313b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:23.507268 6946 tablet_bootstrap.cc:492] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78: Bootstrap starting.
I20260318 08:59:23.508704 6947 tablet_bootstrap.cc:492] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28: Bootstrap starting.
I20260318 08:59:23.511583 6946 tablet_bootstrap.cc:654] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:23.512928 6947 tablet_bootstrap.cc:654] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:23.515756 6946 tablet_bootstrap.cc:492] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78: No bootstrap required, opened a new log
I20260318 08:59:23.516197 6946 ts_tablet_manager.cc:1403] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260318 08:59:23.517175 6947 tablet_bootstrap.cc:492] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28: No bootstrap required, opened a new log
I20260318 08:59:23.517522 6947 ts_tablet_manager.cc:1403] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260318 08:59:23.517984 6946 raft_consensus.cc:359] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e80718d3b4ef0851babd193edca28" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35987 } } peers { permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 } }
I20260318 08:59:23.518461 6946 raft_consensus.cc:740] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a816fb8f841a4f3b82d55d188f475e78, State: Initialized, Role: FOLLOWER
I20260318 08:59:23.518908 6946 consensus_queue.cc:260] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e80718d3b4ef0851babd193edca28" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35987 } } peers { permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 } }
I20260318 08:59:23.519366 6947 raft_consensus.cc:359] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e80718d3b4ef0851babd193edca28" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35987 } } peers { permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 } }
I20260318 08:59:23.519841 6947 raft_consensus.cc:740] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a68e80718d3b4ef0851babd193edca28, State: Initialized, Role: FOLLOWER
I20260318 08:59:23.520308 6947 consensus_queue.cc:260] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e80718d3b4ef0851babd193edca28" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35987 } } peers { permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 } }
I20260318 08:59:23.520910 6933 heartbeater.cc:499] Master 127.6.2.126:37125 was elected leader, sending a full tablet report...
I20260318 08:59:23.521462 6946 ts_tablet_manager.cc:1434] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260318 08:59:23.523608 6947 ts_tablet_manager.cc:1434] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260318 08:59:23.551543 6153 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260318 08:59:23.551797 6153 raft_consensus.cc:423] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 0 FOLLOWER]: Emulating election...
I20260318 08:59:23.552022 6153 raft_consensus.cc:3060] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:23.555840 6153 raft_consensus.cc:697] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 1 LEADER]: Becoming Leader. State: Replica: a68e80718d3b4ef0851babd193edca28, State: Running, Role: LEADER
W20260318 08:59:23.556161 6858 heartbeater.cc:646] Failed to heartbeat to 127.6.2.126:37125 (0 consecutive failures): IO error: failing all heartbeats for tests
I20260318 08:59:23.556504 6153 consensus_queue.cc:237] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e80718d3b4ef0851babd193edca28" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35987 } } peers { permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 } }
I20260318 08:59:23.559238 6153 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260318 08:59:23.575281 6153 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260318 08:59:23.679049 6153 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260318 08:59:23.882659 6153 cluster_itest_util.cc:258] Not converged past 1 yet: 1.1 0.0
I20260318 08:59:24.050509 6908 raft_consensus.cc:3060] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:24.055872 6951 consensus_queue.cc:1048] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 08:59:24.187944 6153 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260318 08:59:24.188156 6153 raft_consensus.cc:423] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 1 LEADER]: Emulating election...
I20260318 08:59:24.188334 6153 raft_consensus.cc:3055] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 1 LEADER]: Stepping down as leader of term 1
I20260318 08:59:24.188529 6153 raft_consensus.cc:740] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a68e80718d3b4ef0851babd193edca28, State: Running, Role: LEADER
I20260318 08:59:24.188891 6153 consensus_queue.cc:260] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e80718d3b4ef0851babd193edca28" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35987 } } peers { permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 } }
I20260318 08:59:24.189473 6153 raft_consensus.cc:3060] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:24.193257 6153 raft_consensus.cc:697] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 2 LEADER]: Becoming Leader. State: Replica: a68e80718d3b4ef0851babd193edca28, State: Running, Role: LEADER
I20260318 08:59:24.193817 6153 consensus_queue.cc:237] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e80718d3b4ef0851babd193edca28" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35987 } } peers { permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 } }
I20260318 08:59:24.195025 6153 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260318 08:59:24.198141 6153 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260318 08:59:24.301856 6153 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260318 08:59:24.505360 6153 cluster_itest_util.cc:258] Not converged past 2 yet: 2.2 1.1
I20260318 08:59:24.697358 6908 raft_consensus.cc:3060] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:24.701853 6908 raft_consensus.cc:1275] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78 [term 2 FOLLOWER]: Refusing update from remote peer a68e80718d3b4ef0851babd193edca28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260318 08:59:24.702900 6960 consensus_queue.cc:1048] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260318 08:59:24.810216 6153 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20260318 08:59:24.810524 6153 raft_consensus.cc:423] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 2 LEADER]: Emulating election...
I20260318 08:59:24.810700 6153 raft_consensus.cc:3055] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 2 LEADER]: Stepping down as leader of term 2
I20260318 08:59:24.810844 6153 raft_consensus.cc:740] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a68e80718d3b4ef0851babd193edca28, State: Running, Role: LEADER
I20260318 08:59:24.811299 6153 consensus_queue.cc:260] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e80718d3b4ef0851babd193edca28" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35987 } } peers { permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 } }
I20260318 08:59:24.811868 6153 raft_consensus.cc:3060] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 2 FOLLOWER]: Advancing to term 3
I20260318 08:59:24.815438 6153 raft_consensus.cc:697] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 3 LEADER]: Becoming Leader. State: Replica: a68e80718d3b4ef0851babd193edca28, State: Running, Role: LEADER
I20260318 08:59:24.815930 6153 consensus_queue.cc:237] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a68e80718d3b4ef0851babd193edca28" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35987 } } peers { permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 } }
I20260318 08:59:24.817104 6153 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20260318 08:59:24.820236 6153 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260318 08:59:24.923925 6153 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260318 08:59:25.127609 6153 cluster_itest_util.cc:258] Not converged past 3 yet: 3.3 2.2
I20260318 08:59:25.291986 6908 raft_consensus.cc:3060] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78 [term 2 FOLLOWER]: Advancing to term 3
I20260318 08:59:25.296312 6908 raft_consensus.cc:1275] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78 [term 3 FOLLOWER]: Refusing update from remote peer a68e80718d3b4ef0851babd193edca28: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260318 08:59:25.297369 6962 consensus_queue.cc:1048] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a816fb8f841a4f3b82d55d188f475e78" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 08:59:25.433554 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 08:59:25.449018 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:25.449527 6153 tablet_replica.cc:333] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28: stopping tablet replica
I20260318 08:59:25.449934 6153 raft_consensus.cc:2243] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 3 LEADER]: Raft consensus shutting down.
I20260318 08:59:25.450417 6153 raft_consensus.cc:2272] T fabc9dccd785425b91bdba4895c3313b P a68e80718d3b4ef0851babd193edca28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:25.467800 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 08:59:25.474576 6153 tablet_server.cc:178] TabletServer@127.6.2.66:0 shutting down...
I20260318 08:59:25.489954 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:25.490465 6153 tablet_replica.cc:333] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78: stopping tablet replica
I20260318 08:59:25.490859 6153 raft_consensus.cc:2243] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260318 08:59:25.491321 6153 raft_consensus.cc:2272] T fabc9dccd785425b91bdba4895c3313b P a816fb8f841a4f3b82d55d188f475e78 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:25.508260 6153 tablet_server.cc:195] TabletServer@127.6.2.66:0 shutdown complete.
I20260318 08:59:25.514760 6153 master.cc:561] Master@127.6.2.126:37125 shutting down...
I20260318 08:59:25.526782 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 1 LEADER]: Raft consensus shutting down.
I20260318 08:59:25.527238 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:25.527513 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7611abd82be64877ae6e5c1bd38798c9: stopping tablet replica
I20260318 08:59:25.543247 6153 master.cc:583] Master@127.6.2.126:37125 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2463 ms)
[----------] 3 tests from LeadershipChangeReportingTest (9971 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20260318 08:59:25.565344 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:39147
I20260318 08:59:25.566325 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:25.570621 6973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:25.571889 6974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:25.572984 6976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:25.573668 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:25.574235 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:25.574404 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:25.574545 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824365574518 us; error 0 us; skew 500 ppm
I20260318 08:59:25.575029 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:25.577054 6153 webserver.cc:492] Webserver started at http://127.6.2.126:44699/ using document root <none> and password file <none>
I20260318 08:59:25.577471 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:25.577665 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:25.577909 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:25.578873 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "34206776523a4de5b0428c08fa74398f"
format_stamp: "Formatted at 2026-03-18 08:59:25 on dist-test-slave-mpvv"
I20260318 08:59:25.582686 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260318 08:59:25.585618 6981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:25.586222 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260318 08:59:25.586508 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "34206776523a4de5b0428c08fa74398f"
format_stamp: "Formatted at 2026-03-18 08:59:25 on dist-test-slave-mpvv"
I20260318 08:59:25.586755 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:25.613827 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:25.615312 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:25.652336 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:39147
I20260318 08:59:25.652498 7033 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:39147 every 8 connection(s)
I20260318 08:59:25.656566 7034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:25.666568 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f: Bootstrap starting.
I20260318 08:59:25.670925 7034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:25.674757 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f: No bootstrap required, opened a new log
I20260318 08:59:25.676779 7034 raft_consensus.cc:359] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34206776523a4de5b0428c08fa74398f" member_type: VOTER }
I20260318 08:59:25.677233 7034 raft_consensus.cc:385] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:25.677489 7034 raft_consensus.cc:740] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34206776523a4de5b0428c08fa74398f, State: Initialized, Role: FOLLOWER
I20260318 08:59:25.678022 7034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34206776523a4de5b0428c08fa74398f" member_type: VOTER }
I20260318 08:59:25.678467 7034 raft_consensus.cc:399] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 08:59:25.678725 7034 raft_consensus.cc:493] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 08:59:25.679015 7034 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:25.682678 7034 raft_consensus.cc:515] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34206776523a4de5b0428c08fa74398f" member_type: VOTER }
I20260318 08:59:25.683096 7034 leader_election.cc:304] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34206776523a4de5b0428c08fa74398f; no voters:
I20260318 08:59:25.684231 7034 leader_election.cc:290] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 08:59:25.684594 7037 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 1 FOLLOWER]: Leader election won for term 1
I20260318 08:59:25.685799 7037 raft_consensus.cc:697] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 1 LEADER]: Becoming Leader. State: Replica: 34206776523a4de5b0428c08fa74398f, State: Running, Role: LEADER
I20260318 08:59:25.686443 7037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34206776523a4de5b0428c08fa74398f" member_type: VOTER }
I20260318 08:59:25.687080 7034 sys_catalog.cc:565] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [sys.catalog]: configured and running, proceeding with master startup.
I20260318 08:59:25.689651 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34206776523a4de5b0428c08fa74398f. Latest consensus state: current_term: 1 leader_uuid: "34206776523a4de5b0428c08fa74398f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34206776523a4de5b0428c08fa74398f" member_type: VOTER } }
I20260318 08:59:25.689680 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34206776523a4de5b0428c08fa74398f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34206776523a4de5b0428c08fa74398f" member_type: VOTER } }
I20260318 08:59:25.690282 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:25.690310 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:25.696666 7043 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 08:59:25.702116 7043 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 08:59:25.703032 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 08:59:25.710036 7043 catalog_manager.cc:1357] Generated new cluster ID: 4598cb05a6cd4593b71cd37076ea6ba1
I20260318 08:59:25.710274 7043 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 08:59:25.726145 7043 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 08:59:25.727207 7043 catalog_manager.cc:1514] Loading token signing keys...
I20260318 08:59:25.737042 7043 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f: Generated new TSK 0
I20260318 08:59:25.737555 7043 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 08:59:25.769121 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:25.775701 7055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:25.776389 7056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:25.778558 7058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:25.779347 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:25.780087 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:25.780313 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:25.780485 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824365780468 us; error 0 us; skew 500 ppm
I20260318 08:59:25.781066 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:25.783360 6153 webserver.cc:492] Webserver started at http://127.6.2.65:45263/ using document root <none> and password file <none>
I20260318 08:59:25.783984 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:25.784298 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:25.784693 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:25.785910 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "c46cc1210a08484da0dbc6dab4a7e145"
format_stamp: "Formatted at 2026-03-18 08:59:25 on dist-test-slave-mpvv"
I20260318 08:59:25.790446 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260318 08:59:25.793721 7063 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:25.794471 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260318 08:59:25.794751 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "c46cc1210a08484da0dbc6dab4a7e145"
format_stamp: "Formatted at 2026-03-18 08:59:25 on dist-test-slave-mpvv"
I20260318 08:59:25.795029 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:25.821687 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:25.822844 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:25.824256 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:25.826535 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:25.826769 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:25.827008 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:25.827219 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260318 08:59:25.866320 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:44271
I20260318 08:59:25.866447 7125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:44271 every 8 connection(s)
I20260318 08:59:25.871302 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:25.879274 7130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:25.880657 7131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:25.883647 7133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:25.884392 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:25.885396 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:25.885640 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:25.885838 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824365885818 us; error 0 us; skew 500 ppm
I20260318 08:59:25.886533 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:25.886744 7126 heartbeater.cc:344] Connected to a master server at 127.6.2.126:39147
I20260318 08:59:25.887001 7126 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:25.887627 7126 heartbeater.cc:507] Master 127.6.2.126:39147 requested a full tablet report, sending...
I20260318 08:59:25.889266 6153 webserver.cc:492] Webserver started at http://127.6.2.66:35425/ using document root <none> and password file <none>
I20260318 08:59:25.889885 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:25.889827 6999 ts_manager.cc:194] Registered new tserver with Master: c46cc1210a08484da0dbc6dab4a7e145 (127.6.2.65:44271)
I20260318 08:59:25.890187 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:25.890516 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:25.891981 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-1-root/instance:
uuid: "c1fd0d81816040fdb677f0092d292f13"
format_stamp: "Formatted at 2026-03-18 08:59:25 on dist-test-slave-mpvv"
I20260318 08:59:25.892196 6999 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40066
I20260318 08:59:25.896648 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260318 08:59:25.899765 7138 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:25.900468 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260318 08:59:25.900722 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-1-root
uuid: "c1fd0d81816040fdb677f0092d292f13"
format_stamp: "Formatted at 2026-03-18 08:59:25 on dist-test-slave-mpvv"
I20260318 08:59:25.900964 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:25.916487 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:25.917546 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:25.918886 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:25.921022 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:25.921243 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:25.921468 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:25.921661 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:25.959525 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.66:33211
I20260318 08:59:25.959610 7200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.66:33211 every 8 connection(s)
I20260318 08:59:25.963881 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:25.970351 7204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:25.973950 7205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:25.976435 7207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:25.976740 7201 heartbeater.cc:344] Connected to a master server at 127.6.2.126:39147
I20260318 08:59:25.977128 7201 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:25.977794 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:25.977982 7201 heartbeater.cc:507] Master 127.6.2.126:39147 requested a full tablet report, sending...
I20260318 08:59:25.978899 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:25.979182 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:25.979388 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824365979368 us; error 0 us; skew 500 ppm
I20260318 08:59:25.980084 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:25.980386 6999 ts_manager.cc:194] Registered new tserver with Master: c1fd0d81816040fdb677f0092d292f13 (127.6.2.66:33211)
I20260318 08:59:25.982398 6999 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40074
I20260318 08:59:25.983322 6153 webserver.cc:492] Webserver started at http://127.6.2.67:33443/ using document root <none> and password file <none>
I20260318 08:59:25.983803 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:25.984009 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:25.984297 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:25.985410 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-2-root/instance:
uuid: "977cb64033274e78a299bf04f4e33c88"
format_stamp: "Formatted at 2026-03-18 08:59:25 on dist-test-slave-mpvv"
I20260318 08:59:25.989866 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260318 08:59:25.992888 7212 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:25.993657 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260318 08:59:25.993935 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-2-root
uuid: "977cb64033274e78a299bf04f4e33c88"
format_stamp: "Formatted at 2026-03-18 08:59:25 on dist-test-slave-mpvv"
I20260318 08:59:25.994204 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1773824355427976-6153-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:26.009953 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:26.011000 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:26.012384 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:26.014429 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:26.014649 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:26.014886 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:26.015051 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:26.078063 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.67:37467
I20260318 08:59:26.078154 7274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.67:37467 every 8 connection(s)
I20260318 08:59:26.093490 7275 heartbeater.cc:344] Connected to a master server at 127.6.2.126:39147
I20260318 08:59:26.093843 7275 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:26.094497 7275 heartbeater.cc:507] Master 127.6.2.126:39147 requested a full tablet report, sending...
I20260318 08:59:26.096372 6999 ts_manager.cc:194] Registered new tserver with Master: 977cb64033274e78a299bf04f4e33c88 (127.6.2.67:37467)
I20260318 08:59:26.096534 6153 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014371115s
I20260318 08:59:26.097782 6999 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40084
I20260318 08:59:26.119009 6999 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260318 08:59:26.120590 6999 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260318 08:59:26.155206 7166 tablet_service.cc:1511] Processing CreateTablet for tablet 3564ab6807504f0b823d9e25a3a3ee78 (DEFAULT_TABLE table=test-table [id=54c26c34d845425ea4c490c157fc2b58]), partition=
I20260318 08:59:26.156622 7166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3564ab6807504f0b823d9e25a3a3ee78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:26.158022 7240 tablet_service.cc:1511] Processing CreateTablet for tablet 3564ab6807504f0b823d9e25a3a3ee78 (DEFAULT_TABLE table=test-table [id=54c26c34d845425ea4c490c157fc2b58]), partition=
I20260318 08:59:26.159060 7240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3564ab6807504f0b823d9e25a3a3ee78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:26.169368 7091 tablet_service.cc:1511] Processing CreateTablet for tablet 3564ab6807504f0b823d9e25a3a3ee78 (DEFAULT_TABLE table=test-table [id=54c26c34d845425ea4c490c157fc2b58]), partition=
I20260318 08:59:26.170596 7091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3564ab6807504f0b823d9e25a3a3ee78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:26.172165 7290 tablet_bootstrap.cc:492] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13: Bootstrap starting.
I20260318 08:59:26.176242 7291 tablet_bootstrap.cc:492] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88: Bootstrap starting.
I20260318 08:59:26.179941 7290 tablet_bootstrap.cc:654] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:26.182996 7291 tablet_bootstrap.cc:654] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:26.188167 7292 tablet_bootstrap.cc:492] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145: Bootstrap starting.
I20260318 08:59:26.188540 7290 tablet_bootstrap.cc:492] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13: No bootstrap required, opened a new log
I20260318 08:59:26.188997 7290 ts_tablet_manager.cc:1403] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20260318 08:59:26.189929 7291 tablet_bootstrap.cc:492] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88: No bootstrap required, opened a new log
I20260318 08:59:26.190323 7291 ts_tablet_manager.cc:1403] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260318 08:59:26.191792 7290 raft_consensus.cc:359] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.192514 7290 raft_consensus.cc:385] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:26.192780 7290 raft_consensus.cc:740] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fd0d81816040fdb677f0092d292f13, State: Initialized, Role: FOLLOWER
I20260318 08:59:26.193158 7292 tablet_bootstrap.cc:654] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:26.192939 7291 raft_consensus.cc:359] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.193600 7291 raft_consensus.cc:385] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:26.193405 7290 consensus_queue.cc:260] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.193866 7291 raft_consensus.cc:740] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977cb64033274e78a299bf04f4e33c88, State: Initialized, Role: FOLLOWER
I20260318 08:59:26.194555 7291 consensus_queue.cc:260] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.200093 7292 tablet_bootstrap.cc:492] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145: No bootstrap required, opened a new log
I20260318 08:59:26.200410 7275 heartbeater.cc:499] Master 127.6.2.126:39147 was elected leader, sending a full tablet report...
I20260318 08:59:26.200589 7292 ts_tablet_manager.cc:1403] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260318 08:59:26.201141 7291 ts_tablet_manager.cc:1434] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260318 08:59:26.202662 7290 ts_tablet_manager.cc:1434] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260318 08:59:26.203256 7201 heartbeater.cc:499] Master 127.6.2.126:39147 was elected leader, sending a full tablet report...
I20260318 08:59:26.203210 7292 raft_consensus.cc:359] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.203879 7292 raft_consensus.cc:385] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:26.204110 7292 raft_consensus.cc:740] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c46cc1210a08484da0dbc6dab4a7e145, State: Initialized, Role: FOLLOWER
I20260318 08:59:26.204643 7292 consensus_queue.cc:260] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.207192 7292 ts_tablet_manager.cc:1434] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260318 08:59:26.207465 7126 heartbeater.cc:499] Master 127.6.2.126:39147 was elected leader, sending a full tablet report...
I20260318 08:59:26.295646 7297 raft_consensus.cc:493] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:26.296300 7297 raft_consensus.cc:515] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.302801 7297 leader_election.cc:290] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1fd0d81816040fdb677f0092d292f13 (127.6.2.66:33211), c46cc1210a08484da0dbc6dab4a7e145 (127.6.2.65:44271)
I20260318 08:59:26.763635 7300 raft_consensus.cc:493] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:26.764426 7300 raft_consensus.cc:515] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.766758 7300 leader_election.cc:290] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 977cb64033274e78a299bf04f4e33c88 (127.6.2.67:37467), c46cc1210a08484da0dbc6dab4a7e145 (127.6.2.65:44271)
I20260318 08:59:26.893756 7306 raft_consensus.cc:493] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:26.894414 7306 raft_consensus.cc:515] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.898236 7250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3564ab6807504f0b823d9e25a3a3ee78" candidate_uuid: "c1fd0d81816040fdb677f0092d292f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "977cb64033274e78a299bf04f4e33c88" is_pre_election: true
I20260318 08:59:26.899163 7250 raft_consensus.cc:2468] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fd0d81816040fdb677f0092d292f13 in term 0.
I20260318 08:59:26.900413 7142 leader_election.cc:304] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 977cb64033274e78a299bf04f4e33c88, c1fd0d81816040fdb677f0092d292f13; no voters:
I20260318 08:59:26.901142 7300 raft_consensus.cc:2804] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260318 08:59:26.901445 7300 raft_consensus.cc:493] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 08:59:26.901767 7300 raft_consensus.cc:3060] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:26.904397 7306 leader_election.cc:290] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1fd0d81816040fdb677f0092d292f13 (127.6.2.66:33211), 977cb64033274e78a299bf04f4e33c88 (127.6.2.67:37467)
I20260318 08:59:26.910815 7300 raft_consensus.cc:515] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.915827 7300 leader_election.cc:290] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [CANDIDATE]: Term 1 election: Requested vote from peers 977cb64033274e78a299bf04f4e33c88 (127.6.2.67:37467), c46cc1210a08484da0dbc6dab4a7e145 (127.6.2.65:44271)
I20260318 08:59:26.915685 7250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3564ab6807504f0b823d9e25a3a3ee78" candidate_uuid: "c1fd0d81816040fdb677f0092d292f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "977cb64033274e78a299bf04f4e33c88"
I20260318 08:59:26.916311 7250 raft_consensus.cc:3060] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:26.924376 7101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3564ab6807504f0b823d9e25a3a3ee78" candidate_uuid: "977cb64033274e78a299bf04f4e33c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c46cc1210a08484da0dbc6dab4a7e145" is_pre_election: true
I20260318 08:59:26.925012 7101 raft_consensus.cc:2468] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 977cb64033274e78a299bf04f4e33c88 in term 0.
I20260318 08:59:26.927613 7215 leader_election.cc:304] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 977cb64033274e78a299bf04f4e33c88, c46cc1210a08484da0dbc6dab4a7e145; no voters:
I20260318 08:59:26.935412 7176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3564ab6807504f0b823d9e25a3a3ee78" candidate_uuid: "977cb64033274e78a299bf04f4e33c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1fd0d81816040fdb677f0092d292f13" is_pre_election: true
I20260318 08:59:26.936229 7176 raft_consensus.cc:2393] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 977cb64033274e78a299bf04f4e33c88 in current term 1: Already voted for candidate c1fd0d81816040fdb677f0092d292f13 in this term.
I20260318 08:59:26.936146 7175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3564ab6807504f0b823d9e25a3a3ee78" candidate_uuid: "c46cc1210a08484da0dbc6dab4a7e145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1fd0d81816040fdb677f0092d292f13" is_pre_election: true
I20260318 08:59:26.936900 7175 raft_consensus.cc:2393] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c46cc1210a08484da0dbc6dab4a7e145 in current term 1: Already voted for candidate c1fd0d81816040fdb677f0092d292f13 in this term.
I20260318 08:59:26.936828 7250 raft_consensus.cc:2468] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fd0d81816040fdb677f0092d292f13 in term 1.
I20260318 08:59:26.939754 7142 leader_election.cc:304] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 977cb64033274e78a299bf04f4e33c88, c1fd0d81816040fdb677f0092d292f13; no voters:
I20260318 08:59:26.940675 7300 raft_consensus.cc:2804] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 08:59:26.940878 7312 raft_consensus.cc:2764] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260318 08:59:26.941617 7101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3564ab6807504f0b823d9e25a3a3ee78" candidate_uuid: "c1fd0d81816040fdb677f0092d292f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c46cc1210a08484da0dbc6dab4a7e145" is_pre_election: true
I20260318 08:59:26.941968 7300 raft_consensus.cc:697] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 1 LEADER]: Becoming Leader. State: Replica: c1fd0d81816040fdb677f0092d292f13, State: Running, Role: LEADER
I20260318 08:59:26.942255 7101 raft_consensus.cc:2468] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fd0d81816040fdb677f0092d292f13 in term 0.
I20260318 08:59:26.942741 7300 consensus_queue.cc:237] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } }
I20260318 08:59:26.943408 7101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3564ab6807504f0b823d9e25a3a3ee78" candidate_uuid: "c1fd0d81816040fdb677f0092d292f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c46cc1210a08484da0dbc6dab4a7e145"
I20260318 08:59:26.943885 7101 raft_consensus.cc:3060] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:26.945744 7250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3564ab6807504f0b823d9e25a3a3ee78" candidate_uuid: "c46cc1210a08484da0dbc6dab4a7e145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "977cb64033274e78a299bf04f4e33c88" is_pre_election: true
I20260318 08:59:26.946344 7250 raft_consensus.cc:2393] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c46cc1210a08484da0dbc6dab4a7e145 in current term 1: Already voted for candidate c1fd0d81816040fdb677f0092d292f13 in this term.
I20260318 08:59:26.947633 7067 leader_election.cc:304] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c46cc1210a08484da0dbc6dab4a7e145; no voters: 977cb64033274e78a299bf04f4e33c88, c1fd0d81816040fdb677f0092d292f13
I20260318 08:59:26.951717 7101 raft_consensus.cc:2468] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fd0d81816040fdb677f0092d292f13 in term 1.
I20260318 08:59:26.952366 7306 raft_consensus.cc:2749] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260318 08:59:26.954661 6998 catalog_manager.cc:5671] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1fd0d81816040fdb677f0092d292f13 (127.6.2.66). New cstate: current_term: 1 leader_uuid: "c1fd0d81816040fdb677f0092d292f13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1fd0d81816040fdb677f0092d292f13" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 } health_report { overall_health: UNKNOWN } } }
I20260318 08:59:27.325716 7300 consensus_queue.cc:1048] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c46cc1210a08484da0dbc6dab4a7e145" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 44271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 08:59:27.336460 7322 consensus_queue.cc:1048] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "977cb64033274e78a299bf04f4e33c88" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260318 08:59:27.774889 7271 debug-util.cc:398] Leaking SignalData structure 0x7b0800173060 after lost signal to thread 6156
I20260318 08:59:28.060806 6153 tablet_replica.cc:333] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145: stopping tablet replica
I20260318 08:59:28.061339 6153 raft_consensus.cc:2243] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 08:59:28.061777 6153 raft_consensus.cc:2272] T 3564ab6807504f0b823d9e25a3a3ee78 P c46cc1210a08484da0dbc6dab4a7e145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:28.418792 7141 consensus_queue.cc:579] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [LEADER]: The tablet replica hosted on peer c46cc1210a08484da0dbc6dab4a7e145 has failed
W20260318 08:59:28.419462 7141 consensus_peers.cc:597] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 -> Peer c46cc1210a08484da0dbc6dab4a7e145 (127.6.2.65:44271): Couldn't send request to peer c46cc1210a08484da0dbc6dab4a7e145. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260318 08:59:28.584108 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 08:59:28.611434 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:28.630508 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 08:59:28.641165 6153 tablet_server.cc:178] TabletServer@127.6.2.66:0 shutting down...
I20260318 08:59:28.661940 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:28.662463 6153 tablet_replica.cc:333] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13: stopping tablet replica
I20260318 08:59:28.662940 6153 raft_consensus.cc:2243] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 1 LEADER]: Raft consensus shutting down.
I20260318 08:59:28.663708 6153 raft_consensus.cc:2272] T 3564ab6807504f0b823d9e25a3a3ee78 P c1fd0d81816040fdb677f0092d292f13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:28.672101 6153 tablet_server.cc:195] TabletServer@127.6.2.66:0 shutdown complete.
I20260318 08:59:28.681989 6153 tablet_server.cc:178] TabletServer@127.6.2.67:0 shutting down...
I20260318 08:59:28.700031 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:28.700604 6153 tablet_replica.cc:333] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88: stopping tablet replica
I20260318 08:59:28.701092 6153 raft_consensus.cc:2243] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 08:59:28.701453 6153 raft_consensus.cc:2272] T 3564ab6807504f0b823d9e25a3a3ee78 P 977cb64033274e78a299bf04f4e33c88 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260318 08:59:29.716922 6153 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 7271)
I20260318 08:59:29.793434 6153 tablet_server.cc:195] TabletServer@127.6.2.67:0 shutdown complete.
I20260318 08:59:29.801225 6153 master.cc:561] Master@127.6.2.126:39147 shutting down...
I20260318 08:59:29.817355 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 1 LEADER]: Raft consensus shutting down.
I20260318 08:59:29.817853 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:29.818157 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 34206776523a4de5b0428c08fa74398f: stopping tablet replica
I20260318 08:59:29.835024 6153 master.cc:583] Master@127.6.2.126:39147 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (4298 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20260318 08:59:29.864871 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:43831
I20260318 08:59:29.865947 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:29.870565 7332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:29.872197 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:29.873008 7335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:29.873991 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:29.874775 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:29.874971 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:29.875109 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824369875090 us; error 0 us; skew 500 ppm
I20260318 08:59:29.875644 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:29.877774 6153 webserver.cc:492] Webserver started at http://127.6.2.126:33379/ using document root <none> and password file <none>
I20260318 08:59:29.878286 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:29.878445 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:29.878669 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:29.879727 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "ae68aa75f65a4c0a87e5a4022c4bdd5e"
format_stamp: "Formatted at 2026-03-18 08:59:29 on dist-test-slave-mpvv"
I20260318 08:59:29.884086 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260318 08:59:29.887501 7340 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:29.888235 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260318 08:59:29.888530 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "ae68aa75f65a4c0a87e5a4022c4bdd5e"
format_stamp: "Formatted at 2026-03-18 08:59:29 on dist-test-slave-mpvv"
I20260318 08:59:29.888808 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:29.913724 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:29.915014 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:29.949664 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:43831
I20260318 08:59:29.949766 7391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:43831 every 8 connection(s)
I20260318 08:59:29.953879 7392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:29.963722 7392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e: Bootstrap starting.
I20260318 08:59:29.967878 7392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:29.971861 7392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e: No bootstrap required, opened a new log
I20260318 08:59:29.974287 7392 raft_consensus.cc:359] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae68aa75f65a4c0a87e5a4022c4bdd5e" member_type: VOTER }
I20260318 08:59:29.974882 7392 raft_consensus.cc:385] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:29.975224 7392 raft_consensus.cc:740] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae68aa75f65a4c0a87e5a4022c4bdd5e, State: Initialized, Role: FOLLOWER
I20260318 08:59:29.975963 7392 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae68aa75f65a4c0a87e5a4022c4bdd5e" member_type: VOTER }
I20260318 08:59:29.976657 7392 raft_consensus.cc:399] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 08:59:29.976995 7392 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 08:59:29.977365 7392 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:29.983512 7392 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae68aa75f65a4c0a87e5a4022c4bdd5e" member_type: VOTER }
I20260318 08:59:29.984284 7392 leader_election.cc:304] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae68aa75f65a4c0a87e5a4022c4bdd5e; no voters:
I20260318 08:59:29.985559 7392 leader_election.cc:290] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 08:59:29.985957 7395 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 1 FOLLOWER]: Leader election won for term 1
I20260318 08:59:29.987414 7395 raft_consensus.cc:697] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 1 LEADER]: Becoming Leader. State: Replica: ae68aa75f65a4c0a87e5a4022c4bdd5e, State: Running, Role: LEADER
I20260318 08:59:29.987998 7395 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae68aa75f65a4c0a87e5a4022c4bdd5e" member_type: VOTER }
I20260318 08:59:29.988576 7392 sys_catalog.cc:565] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [sys.catalog]: configured and running, proceeding with master startup.
I20260318 08:59:29.993256 7397 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae68aa75f65a4c0a87e5a4022c4bdd5e. Latest consensus state: current_term: 1 leader_uuid: "ae68aa75f65a4c0a87e5a4022c4bdd5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae68aa75f65a4c0a87e5a4022c4bdd5e" member_type: VOTER } }
I20260318 08:59:29.993355 7396 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae68aa75f65a4c0a87e5a4022c4bdd5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae68aa75f65a4c0a87e5a4022c4bdd5e" member_type: VOTER } }
I20260318 08:59:29.994334 7397 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:29.994391 7396 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:30.002552 7403 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 08:59:30.009807 7403 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 08:59:30.010800 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 08:59:30.018203 7403 catalog_manager.cc:1357] Generated new cluster ID: 914ef35cad0f4fbda16227309463aea1
I20260318 08:59:30.018488 7403 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 08:59:30.043655 7403 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 08:59:30.044937 7403 catalog_manager.cc:1514] Loading token signing keys...
I20260318 08:59:30.055814 7403 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e: Generated new TSK 0
I20260318 08:59:30.056463 7403 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 08:59:30.077644 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:30.083549 7413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:30.084687 7414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.086719 6153 server_base.cc:1060] running on GCE node
W20260318 08:59:30.086615 7416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.087750 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:30.087980 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:30.088158 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824370088130 us; error 0 us; skew 500 ppm
I20260318 08:59:30.088706 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:30.091109 6153 webserver.cc:492] Webserver started at http://127.6.2.65:35459/ using document root <none> and password file <none>
I20260318 08:59:30.091681 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:30.091892 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:30.092162 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:30.093283 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "7b36bd20f93e42ee8cdfae1cad204981"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.097790 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260318 08:59:30.101222 7421 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.101949 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260318 08:59:30.102234 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "7b36bd20f93e42ee8cdfae1cad204981"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.102571 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:30.119920 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:30.121054 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:30.122469 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:30.124773 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:30.124994 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.125236 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:30.125411 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.163738 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:39459
I20260318 08:59:30.163825 7483 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:39459 every 8 connection(s)
I20260318 08:59:30.177868 7484 heartbeater.cc:344] Connected to a master server at 127.6.2.126:43831
I20260318 08:59:30.178254 7484 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:30.178966 7484 heartbeater.cc:507] Master 127.6.2.126:43831 requested a full tablet report, sending...
I20260318 08:59:30.180832 7357 ts_manager.cc:194] Registered new tserver with Master: 7b36bd20f93e42ee8cdfae1cad204981 (127.6.2.65:39459)
I20260318 08:59:30.180945 6153 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.014188524s
I20260318 08:59:30.182530 7357 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57484
I20260318 08:59:30.200558 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 08:59:30.215349 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:30.230621 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 08:59:30.237924 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260318 08:59:30.240649 6153 tablet_server.cc:93] deduplicated master addresses: 127.6.2.126:43831
W20260318 08:59:30.243721 7494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:30.244134 7495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:30.246133 7497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.246862 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:30.247686 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:30.247862 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:30.248039 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824370248025 us; error 0 us; skew 500 ppm
I20260318 08:59:30.248582 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:30.250834 6153 webserver.cc:492] Webserver started at http://127.6.2.65:35459/ using document root <none> and password file <none>
I20260318 08:59:30.251382 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:30.251575 6153 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260318 08:59:30.254937 6153 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260318 08:59:30.257619 7502 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.258486 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260318 08:59:30.258718 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "7b36bd20f93e42ee8cdfae1cad204981"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.258960 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:30.285450 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:30.287071 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:30.289098 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:30.291637 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:30.291877 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.292114 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:30.292289 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.331504 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:39459
I20260318 08:59:30.331590 7564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:39459 every 8 connection(s)
I20260318 08:59:30.337826 6153 tablet_server.cc:178] TabletServer@127.6.2.65:39459 shutting down...
I20260318 08:59:30.356098 7565 heartbeater.cc:344] Connected to a master server at 127.6.2.126:43831
I20260318 08:59:30.356976 7565 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:30.357688 7565 heartbeater.cc:507] Master 127.6.2.126:43831 requested a full tablet report, sending...
I20260318 08:59:30.359967 7357 ts_manager.cc:194] Re-registered known tserver with Master: 7b36bd20f93e42ee8cdfae1cad204981 (127.6.2.65:39459)
I20260318 08:59:30.361416 7357 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57502
I20260318 08:59:30.363624 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 08:59:30.378786 6153 tablet_server.cc:195] TabletServer@127.6.2.65:39459 shutdown complete.
I20260318 08:59:30.385840 6153 master.cc:561] Master@127.6.2.126:43831 shutting down...
I20260318 08:59:30.398891 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 1 LEADER]: Raft consensus shutting down.
I20260318 08:59:30.399480 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 08:59:30.399839 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P ae68aa75f65a4c0a87e5a4022c4bdd5e: stopping tablet replica
I20260318 08:59:30.418244 6153 master.cc:583] Master@127.6.2.126:43831 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (575 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20260318 08:59:30.440649 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:40111,127.6.2.125:34659,127.6.2.124:42683
I20260318 08:59:30.441970 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:30.446708 7572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:30.447687 7573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:30.451864 7575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.452196 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:30.453043 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:30.453217 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:30.453339 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824370453322 us; error 0 us; skew 500 ppm
I20260318 08:59:30.453801 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:30.455982 6153 webserver.cc:492] Webserver started at http://127.6.2.126:35431/ using document root <none> and password file <none>
I20260318 08:59:30.456421 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:30.456593 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:30.456796 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:30.457827 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "e45b189483e74bc089b2963727d99490"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.462162 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260318 08:59:30.465440 7580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.466203 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260318 08:59:30.466496 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "e45b189483e74bc089b2963727d99490"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.466751 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:30.484946 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:30.485971 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:30.520833 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:40111
I20260318 08:59:30.520920 7631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:40111 every 8 connection(s)
I20260318 08:59:30.524799 7632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:30.524905 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:30.530263 7634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:30.532240 7635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.539036 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:30.540274 7632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } has no permanent_uuid. Determining permanent_uuid...
W20260318 08:59:30.541101 7637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.542018 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:30.542349 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:30.542559 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824370542535 us; error 0 us; skew 500 ppm
I20260318 08:59:30.543078 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:30.549700 6153 webserver.cc:492] Webserver started at http://127.6.2.125:46041/ using document root <none> and password file <none>
I20260318 08:59:30.550240 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:30.550479 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:30.550776 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:30.551980 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-1-root/instance:
uuid: "07f47abe644e4c6786bd63c1251a0ac2"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.557380 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260318 08:59:30.562579 7644 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.563613 6153 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260318 08:59:30.563908 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-1-root
uuid: "07f47abe644e4c6786bd63c1251a0ac2"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.564208 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:30.564891 7632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } has no permanent_uuid. Determining permanent_uuid...
W20260318 08:59:30.567262 7584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.2.125:34659: connect: Connection refused (error 111)
W20260318 08:59:30.570878 7632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.2.125:34659: Network error: Client connection negotiation failed: client connection to 127.6.2.125:34659: connect: Connection refused (error 111)
I20260318 08:59:30.589660 7632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } attempt: 1
W20260318 08:59:30.594076 7632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.2.125:34659: Network error: Client connection negotiation failed: client connection to 127.6.2.125:34659: connect: Connection refused (error 111)
I20260318 08:59:30.602339 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:30.603400 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:30.639349 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.125:34659
I20260318 08:59:30.639438 7696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.125:34659 every 8 connection(s)
I20260318 08:59:30.642952 7697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:30.643136 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:30.648357 7699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.648567 7632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } attempt: 2
W20260318 08:59:30.650003 7700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.650271 7697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } has no permanent_uuid. Determining permanent_uuid...
W20260318 08:59:30.653081 7702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.655035 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:30.655880 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:30.656132 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:30.656334 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824370656312 us; error 0 us; skew 500 ppm
I20260318 08:59:30.657054 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:30.660259 6153 webserver.cc:492] Webserver started at http://127.6.2.124:46443/ using document root <none> and password file <none>
I20260318 08:59:30.660894 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:30.661129 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:30.661479 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:30.662883 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-2-root/instance:
uuid: "1e52aab3abc24498a28a69e7c89452f9"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.665246 7632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } has no permanent_uuid. Determining permanent_uuid...
I20260318 08:59:30.668108 7697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } has no permanent_uuid. Determining permanent_uuid...
I20260318 08:59:30.669123 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
W20260318 08:59:30.671422 7632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.2.124:42683: Network error: Client connection negotiation failed: client connection to 127.6.2.124:42683: connect: Connection refused (error 111)
I20260318 08:59:30.673759 7709 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.674690 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260318 08:59:30.675024 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-2-root
uuid: "1e52aab3abc24498a28a69e7c89452f9"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.675414 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:30.679589 7697 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } has no permanent_uuid. Determining permanent_uuid...
W20260318 08:59:30.685325 7697 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.2.124:42683: Network error: Client connection negotiation failed: client connection to 127.6.2.124:42683: connect: Connection refused (error 111)
I20260318 08:59:30.695870 7632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } attempt: 1
W20260318 08:59:30.700433 7632 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.2.124:42683: Network error: Client connection negotiation failed: client connection to 127.6.2.124:42683: connect: Connection refused (error 111)
I20260318 08:59:30.701800 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:30.702888 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:30.720831 7697 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } attempt: 1
W20260318 08:59:30.725540 7697 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.2.124:42683: Network error: Client connection negotiation failed: client connection to 127.6.2.124:42683: connect: Connection refused (error 111)
I20260318 08:59:30.740913 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.124:42683
I20260318 08:59:30.741025 7761 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.124:42683 every 8 connection(s)
I20260318 08:59:30.744102 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 08:59:30.744808 7762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:30.749912 7632 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } attempt: 2
I20260318 08:59:30.751631 7762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } has no permanent_uuid. Determining permanent_uuid...
I20260318 08:59:30.765184 7762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } has no permanent_uuid. Determining permanent_uuid...
I20260318 08:59:30.769821 7632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490: Bootstrap starting.
I20260318 08:59:30.775444 7632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:30.775380 7762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } has no permanent_uuid. Determining permanent_uuid...
I20260318 08:59:30.780246 7632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490: No bootstrap required, opened a new log
I20260318 08:59:30.782655 7632 raft_consensus.cc:359] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:30.783368 7632 raft_consensus.cc:385] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:30.783694 7632 raft_consensus.cc:740] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45b189483e74bc089b2963727d99490, State: Initialized, Role: FOLLOWER
I20260318 08:59:30.784381 7632 consensus_queue.cc:260] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:30.786088 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:30.786625 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:30.787941 7632 sys_catalog.cc:565] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: configured and running, proceeding with master startup.
I20260318 08:59:30.793310 7762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9: Bootstrap starting.
I20260318 08:59:30.800413 7762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:30.801263 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20260318 08:59:30.801134 7697 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } attempt: 2
W20260318 08:59:30.805243 7778 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260318 08:59:30.805644 7778 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260318 08:59:30.805860 7762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9: No bootstrap required, opened a new log
I20260318 08:59:30.808240 7762 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:30.808954 7762 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:30.809254 7762 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e52aab3abc24498a28a69e7c89452f9, State: Initialized, Role: FOLLOWER
I20260318 08:59:30.809918 7762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:30.812111 7780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:30.812676 7780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:30.813807 7762 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260318 08:59:30.820376 7697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2: Bootstrap starting.
I20260318 08:59:30.827431 7697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2: Neither blocks nor log segments found. Creating new log.
W20260318 08:59:30.830227 7791 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260318 08:59:30.830634 7791 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260318 08:59:30.831820 7697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2: No bootstrap required, opened a new log
I20260318 08:59:30.832616 7767 raft_consensus.cc:493] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:30.833004 7767 raft_consensus.cc:515] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:30.833920 7697 raft_consensus.cc:359] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:30.834551 7697 raft_consensus.cc:385] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:30.834609 7767 leader_election.cc:290] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07f47abe644e4c6786bd63c1251a0ac2 (127.6.2.125:34659), 1e52aab3abc24498a28a69e7c89452f9 (127.6.2.124:42683)
I20260318 08:59:30.834973 7697 raft_consensus.cc:740] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f47abe644e4c6786bd63c1251a0ac2, State: Initialized, Role: FOLLOWER
I20260318 08:59:30.835777 7737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e45b189483e74bc089b2963727d99490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e52aab3abc24498a28a69e7c89452f9" is_pre_election: true
I20260318 08:59:30.836333 7737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e45b189483e74bc089b2963727d99490 in term 0.
I20260318 08:59:30.835799 7697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:30.837334 7581 leader_election.cc:304] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e52aab3abc24498a28a69e7c89452f9, e45b189483e74bc089b2963727d99490; no voters:
I20260318 08:59:30.838299 7767 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260318 08:59:30.838097 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:30.838654 7767 raft_consensus.cc:493] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 08:59:30.838901 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:30.839011 7767 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:30.839854 7697 sys_catalog.cc:565] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260318 08:59:30.839890 7672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e45b189483e74bc089b2963727d99490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f47abe644e4c6786bd63c1251a0ac2" is_pre_election: true
I20260318 08:59:30.840623 7672 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e45b189483e74bc089b2963727d99490 in term 0.
I20260318 08:59:30.844770 7767 raft_consensus.cc:515] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:30.846467 7767 leader_election.cc:290] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [CANDIDATE]: Term 1 election: Requested vote from peers 07f47abe644e4c6786bd63c1251a0ac2 (127.6.2.125:34659), 1e52aab3abc24498a28a69e7c89452f9 (127.6.2.124:42683)
I20260318 08:59:30.847219 7672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e45b189483e74bc089b2963727d99490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f47abe644e4c6786bd63c1251a0ac2"
I20260318 08:59:30.847819 7672 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:30.847815 7737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e45b189483e74bc089b2963727d99490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e52aab3abc24498a28a69e7c89452f9"
I20260318 08:59:30.848495 7737 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:30.854466 7672 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e45b189483e74bc089b2963727d99490 in term 1.
I20260318 08:59:30.855767 7584 leader_election.cc:304] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f47abe644e4c6786bd63c1251a0ac2, e45b189483e74bc089b2963727d99490; no voters:
I20260318 08:59:30.856202 7737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e45b189483e74bc089b2963727d99490 in term 1.
I20260318 08:59:30.856758 7767 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 08:59:30.857692 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20260318 08:59:30.858352 7767 raft_consensus.cc:697] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 1 LEADER]: Becoming Leader. State: Replica: e45b189483e74bc089b2963727d99490, State: Running, Role: LEADER
I20260318 08:59:30.859200 7767 consensus_queue.cc:237] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:30.859575 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 08:59:30.861728 7804 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
E20260318 08:59:30.861893 6153 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20260318 08:59:30.862066 7804 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260318 08:59:30.863523 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e45b189483e74bc089b2963727d99490. Latest consensus state: current_term: 1 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:30.864264 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:30.866235 7807 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
W20260318 08:59:30.869299 7808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:30.870482 7809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.874567 7807 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 08:59:30.874809 6153 server_base.cc:1060] running on GCE node
W20260318 08:59:30.875628 7811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:30.876624 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:30.876935 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:30.877136 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824370877119 us; error 0 us; skew 500 ppm
I20260318 08:59:30.877835 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:30.880839 6153 webserver.cc:492] Webserver started at http://127.6.2.65:43741/ using document root <none> and password file <none>
I20260318 08:59:30.881352 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:30.881567 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:30.881883 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:30.883131 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "06af5a588953450dbb0862d99ddf80f0"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.883742 7737 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 1 FOLLOWER]: Refusing update from remote peer e45b189483e74bc089b2963727d99490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260318 08:59:30.884181 7672 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 1 FOLLOWER]: Refusing update from remote peer e45b189483e74bc089b2963727d99490: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260318 08:59:30.885031 7767 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 08:59:30.885675 7805 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 08:59:30.889108 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260318 08:59:30.894369 7819 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.894397 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e45b189483e74bc089b2963727d99490. Latest consensus state: current_term: 1 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:30.895499 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:30.896068 6153 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260318 08:59:30.896560 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "06af5a588953450dbb0862d99ddf80f0"
format_stamp: "Formatted at 2026-03-18 08:59:30 on dist-test-slave-mpvv"
I20260318 08:59:30.897037 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:30.900333 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:30.901093 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:30.902400 7807 catalog_manager.cc:1357] Generated new cluster ID: c582dd9307024b9483bfbf805eda4259
I20260318 08:59:30.902755 7807 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 08:59:30.928946 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:30.930042 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:30.931885 7780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e45b189483e74bc089b2963727d99490. Latest consensus state: current_term: 1 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:30.932654 7780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:30.935858 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:30.936765 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:30.940987 7780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:30.941711 7780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:30.949870 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:30.951237 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:30.953085 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:30.957029 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:30.957302 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.957612 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:30.957814 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:30.962910 7807 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 08:59:30.964548 7807 catalog_manager.cc:1514] Loading token signing keys...
I20260318 08:59:30.986645 7807 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490: Generated new TSK 0
I20260318 08:59:30.987371 7807 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 08:59:31.021869 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:35351
I20260318 08:59:31.021934 7886 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:35351 every 8 connection(s)
I20260318 08:59:31.036806 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
E20260318 08:59:31.039547 6153 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260318 08:59:31.054455 7887 heartbeater.cc:344] Connected to a master server at 127.6.2.124:42683
I20260318 08:59:31.054960 7887 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:31.055027 7888 heartbeater.cc:344] Connected to a master server at 127.6.2.125:34659
I20260318 08:59:31.055531 7888 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:31.056113 7887 heartbeater.cc:507] Master 127.6.2.124:42683 requested a full tablet report, sending...
I20260318 08:59:31.056406 7888 heartbeater.cc:507] Master 127.6.2.125:34659 requested a full tablet report, sending...
I20260318 08:59:31.057961 7889 heartbeater.cc:344] Connected to a master server at 127.6.2.126:40111
I20260318 08:59:31.058396 7889 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:31.059424 7889 heartbeater.cc:507] Master 127.6.2.126:40111 requested a full tablet report, sending...
I20260318 08:59:31.059995 7662 ts_manager.cc:194] Registered new tserver with Master: 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351)
I20260318 08:59:31.060001 7727 ts_manager.cc:194] Registered new tserver with Master: 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351)
W20260318 08:59:31.061100 7894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:31.062517 7597 ts_manager.cc:194] Registered new tserver with Master: 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351)
W20260318 08:59:31.063001 7895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:31.065737 7597 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35744
I20260318 08:59:31.066735 6153 server_base.cc:1060] running on GCE node
W20260318 08:59:31.067564 7897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:31.068681 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:31.068940 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:31.069154 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824371069138 us; error 0 us; skew 500 ppm
I20260318 08:59:31.069643 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:31.072024 6153 webserver.cc:492] Webserver started at http://127.6.2.66:35705/ using document root <none> and password file <none>
I20260318 08:59:31.072464 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:31.072620 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:31.072815 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:31.073792 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-1-root/instance:
uuid: "3d02c582e0d940cdbd7b66d2c951c9e1"
format_stamp: "Formatted at 2026-03-18 08:59:31 on dist-test-slave-mpvv"
I20260318 08:59:31.078855 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260318 08:59:31.082743 7902 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:31.083568 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260318 08:59:31.083825 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-1-root
uuid: "3d02c582e0d940cdbd7b66d2c951c9e1"
format_stamp: "Formatted at 2026-03-18 08:59:31 on dist-test-slave-mpvv"
I20260318 08:59:31.084069 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:31.108850 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:31.110023 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:31.111938 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:31.114629 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:31.114886 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:31.115182 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:31.115396 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260318 08:59:31.169301 7889 heartbeater.cc:499] Master 127.6.2.126:40111 was elected leader, sending a full tablet report...
I20260318 08:59:31.193248 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.66:45207
I20260318 08:59:31.193356 7964 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.66:45207 every 8 connection(s)
I20260318 08:59:31.216060 7965 heartbeater.cc:344] Connected to a master server at 127.6.2.124:42683
I20260318 08:59:31.216544 7965 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:31.217196 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260318 08:59:31.217542 7965 heartbeater.cc:507] Master 127.6.2.124:42683 requested a full tablet report, sending...
E20260318 08:59:31.219652 6153 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
I20260318 08:59:31.220117 7966 heartbeater.cc:344] Connected to a master server at 127.6.2.125:34659
I20260318 08:59:31.220549 7966 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:31.220465 7727 ts_manager.cc:194] Registered new tserver with Master: 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:31.221614 7966 heartbeater.cc:507] Master 127.6.2.125:34659 requested a full tablet report, sending...
I20260318 08:59:31.224434 7662 ts_manager.cc:194] Registered new tserver with Master: 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
W20260318 08:59:31.226708 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 08:59:31.229687 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:31.231540 7968 heartbeater.cc:344] Connected to a master server at 127.6.2.126:40111
I20260318 08:59:31.232139 7968 heartbeater.cc:461] Registering TS with master...
W20260318 08:59:31.232415 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 08:59:31.233191 7968 heartbeater.cc:507] Master 127.6.2.126:40111 requested a full tablet report, sending...
I20260318 08:59:31.233387 6153 server_base.cc:1060] running on GCE node
I20260318 08:59:31.234488 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 08:59:31.234781 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 08:59:31.234980 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824371234961 us; error 0 us; skew 500 ppm
I20260318 08:59:31.235584 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 08:59:31.235590 7597 ts_manager.cc:194] Registered new tserver with Master: 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:31.237273 7597 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35750
I20260318 08:59:31.238564 6153 webserver.cc:492] Webserver started at http://127.6.2.67:33147/ using document root <none> and password file <none>
I20260318 08:59:31.239234 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 08:59:31.239463 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 08:59:31.239745 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 08:59:31.241212 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-2-root/instance:
uuid: "41e4ea383dc24ae5a6ef557c49dff836"
format_stamp: "Formatted at 2026-03-18 08:59:31 on dist-test-slave-mpvv"
I20260318 08:59:31.245702 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260318 08:59:31.249307 7980 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:31.250043 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260318 08:59:31.250368 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-2-root
uuid: "41e4ea383dc24ae5a6ef557c49dff836"
format_stamp: "Formatted at 2026-03-18 08:59:31 on dist-test-slave-mpvv"
I20260318 08:59:31.250669 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 08:59:31.264097 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 08:59:31.265363 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 08:59:31.266657 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 08:59:31.270061 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 08:59:31.270329 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:31.270614 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 08:59:31.270839 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 08:59:31.339962 7968 heartbeater.cc:499] Master 127.6.2.126:40111 was elected leader, sending a full tablet report...
I20260318 08:59:31.345986 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.67:45185
I20260318 08:59:31.346087 8042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.67:45185 every 8 connection(s)
I20260318 08:59:31.369350 8044 heartbeater.cc:344] Connected to a master server at 127.6.2.125:34659
I20260318 08:59:31.369824 8044 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:31.370817 8044 heartbeater.cc:507] Master 127.6.2.125:34659 requested a full tablet report, sending...
I20260318 08:59:31.371209 8045 heartbeater.cc:344] Connected to a master server at 127.6.2.126:40111
I20260318 08:59:31.372076 8045 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:31.372990 8045 heartbeater.cc:507] Master 127.6.2.126:40111 requested a full tablet report, sending...
I20260318 08:59:31.373579 7662 ts_manager.cc:194] Registered new tserver with Master: 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:31.375201 7597 ts_manager.cc:194] Registered new tserver with Master: 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:31.376502 8043 heartbeater.cc:344] Connected to a master server at 127.6.2.124:42683
I20260318 08:59:31.376925 8043 heartbeater.cc:461] Registering TS with master...
I20260318 08:59:31.377064 7597 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35764
I20260318 08:59:31.377877 8043 heartbeater.cc:507] Master 127.6.2.124:42683 requested a full tablet report, sending...
I20260318 08:59:31.380501 7727 ts_manager.cc:194] Registered new tserver with Master: 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:31.381325 6153 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.02395881s
I20260318 08:59:31.425184 7597 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20260318 08:59:31.427050 7597 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260318 08:59:31.480062 8045 heartbeater.cc:499] Master 127.6.2.126:40111 was elected leader, sending a full tablet report...
I20260318 08:59:31.512485 8008 tablet_service.cc:1511] Processing CreateTablet for tablet 0f3811d2d92f41c2b7301baa5ab3f1cb (DEFAULT_TABLE table=test-table [id=81693cd942f64504990e0adaa14b87f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:31.513010 8007 tablet_service.cc:1511] Processing CreateTablet for tablet 5adac9d378be4afdbe6723195ec9bd11 (DEFAULT_TABLE table=test-table [id=81693cd942f64504990e0adaa14b87f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:31.514034 8008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f3811d2d92f41c2b7301baa5ab3f1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:31.516093 8006 tablet_service.cc:1511] Processing CreateTablet for tablet a936692bfd8f4eaa9b58cb7cf64d05d4 (DEFAULT_TABLE table=test-table [id=81693cd942f64504990e0adaa14b87f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:31.516901 7930 tablet_service.cc:1511] Processing CreateTablet for tablet 0f3811d2d92f41c2b7301baa5ab3f1cb (DEFAULT_TABLE table=test-table [id=81693cd942f64504990e0adaa14b87f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:31.517653 8006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a936692bfd8f4eaa9b58cb7cf64d05d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:31.517349 7929 tablet_service.cc:1511] Processing CreateTablet for tablet 5adac9d378be4afdbe6723195ec9bd11 (DEFAULT_TABLE table=test-table [id=81693cd942f64504990e0adaa14b87f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:31.517946 7928 tablet_service.cc:1511] Processing CreateTablet for tablet a936692bfd8f4eaa9b58cb7cf64d05d4 (DEFAULT_TABLE table=test-table [id=81693cd942f64504990e0adaa14b87f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:31.518399 7930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f3811d2d92f41c2b7301baa5ab3f1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:31.519927 8007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5adac9d378be4afdbe6723195ec9bd11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:31.520339 7852 tablet_service.cc:1511] Processing CreateTablet for tablet 0f3811d2d92f41c2b7301baa5ab3f1cb (DEFAULT_TABLE table=test-table [id=81693cd942f64504990e0adaa14b87f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:31.521319 7929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5adac9d378be4afdbe6723195ec9bd11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:31.523424 7852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f3811d2d92f41c2b7301baa5ab3f1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:31.523765 7850 tablet_service.cc:1511] Processing CreateTablet for tablet a936692bfd8f4eaa9b58cb7cf64d05d4 (DEFAULT_TABLE table=test-table [id=81693cd942f64504990e0adaa14b87f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:31.523694 7851 tablet_service.cc:1511] Processing CreateTablet for tablet 5adac9d378be4afdbe6723195ec9bd11 (DEFAULT_TABLE table=test-table [id=81693cd942f64504990e0adaa14b87f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260318 08:59:31.526966 7850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a936692bfd8f4eaa9b58cb7cf64d05d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:31.525780 7928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a936692bfd8f4eaa9b58cb7cf64d05d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:31.530207 7851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5adac9d378be4afdbe6723195ec9bd11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 08:59:31.576102 8066 tablet_bootstrap.cc:492] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1: Bootstrap starting.
I20260318 08:59:31.576174 8065 tablet_bootstrap.cc:492] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0: Bootstrap starting.
I20260318 08:59:31.626057 8067 tablet_bootstrap.cc:492] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836: Bootstrap starting.
I20260318 08:59:31.630646 8066 tablet_bootstrap.cc:654] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1: Neither blocks nor log segments found. Creating new log.
W20260318 08:59:31.634773 7583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.6.2.67:45185, user_credentials={real_user=slave}} blocked reactor thread for 54461.8us
W20260318 08:59:31.638547 7581 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.6.2.66:45207, user_credentials={real_user=slave}} blocked reactor thread for 47180.1us
I20260318 08:59:31.692067 8065 tablet_bootstrap.cc:654] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:31.694352 8067 tablet_bootstrap.cc:654] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:31.697513 8065 tablet_bootstrap.cc:492] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0: No bootstrap required, opened a new log
I20260318 08:59:31.697970 8065 ts_tablet_manager.cc:1403] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0: Time spent bootstrapping tablet: real 0.123s user 0.016s sys 0.074s
I20260318 08:59:31.700836 8065 raft_consensus.cc:359] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:31.701558 8065 raft_consensus.cc:385] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:31.701850 8065 raft_consensus.cc:740] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06af5a588953450dbb0862d99ddf80f0, State: Initialized, Role: FOLLOWER
I20260318 08:59:31.702551 8065 consensus_queue.cc:260] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
W20260318 08:59:31.703706 7583 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.6.2.67:45185, user_credentials={real_user=slave}} blocked reactor thread for 60364.6us
I20260318 08:59:31.706537 8065 ts_tablet_manager.cc:1434] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0: Time spent starting tablet: real 0.008s user 0.008s sys 0.002s
I20260318 08:59:31.711479 8066 tablet_bootstrap.cc:492] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1: No bootstrap required, opened a new log
I20260318 08:59:31.711560 8065 tablet_bootstrap.cc:492] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0: Bootstrap starting.
I20260318 08:59:31.712091 8066 ts_tablet_manager.cc:1403] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1: Time spent bootstrapping tablet: real 0.137s user 0.023s sys 0.045s
I20260318 08:59:31.715134 8066 raft_consensus.cc:359] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.715950 8066 raft_consensus.cc:385] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:31.716324 8066 raft_consensus.cc:740] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d02c582e0d940cdbd7b66d2c951c9e1, State: Initialized, Role: FOLLOWER
I20260318 08:59:31.717012 8066 consensus_queue.cc:260] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.718555 8065 tablet_bootstrap.cc:654] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:31.732688 8066 ts_tablet_manager.cc:1434] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1: Time spent starting tablet: real 0.020s user 0.011s sys 0.006s
I20260318 08:59:31.736230 8069 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:31.736725 8066 tablet_bootstrap.cc:492] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1: Bootstrap starting.
I20260318 08:59:31.737275 8067 tablet_bootstrap.cc:492] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836: No bootstrap required, opened a new log
I20260318 08:59:31.736747 8069 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:31.737761 8067 ts_tablet_manager.cc:1403] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836: Time spent bootstrapping tablet: real 0.112s user 0.006s sys 0.010s
I20260318 08:59:31.744583 8066 tablet_bootstrap.cc:654] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:31.747246 8065 tablet_bootstrap.cc:492] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0: No bootstrap required, opened a new log
I20260318 08:59:31.747920 8065 ts_tablet_manager.cc:1403] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.011s
I20260318 08:59:31.750928 8065 raft_consensus.cc:359] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.752905 8065 raft_consensus.cc:385] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:31.753314 8065 raft_consensus.cc:740] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06af5a588953450dbb0862d99ddf80f0, State: Initialized, Role: FOLLOWER
I20260318 08:59:31.751394 8069 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:31.751991 8067 raft_consensus.cc:359] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.754045 8065 consensus_queue.cc:260] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.754770 8067 raft_consensus.cc:385] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:31.755350 8067 raft_consensus.cc:740] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e4ea383dc24ae5a6ef557c49dff836, State: Initialized, Role: FOLLOWER
I20260318 08:59:31.756515 8067 consensus_queue.cc:260] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.764088 8065 ts_tablet_manager.cc:1434] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260318 08:59:31.768025 8065 tablet_bootstrap.cc:492] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0: Bootstrap starting.
I20260318 08:59:31.774503 8065 tablet_bootstrap.cc:654] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:31.785986 8067 ts_tablet_manager.cc:1434] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836: Time spent starting tablet: real 0.048s user 0.004s sys 0.008s
I20260318 08:59:31.787917 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:31.788269 8078 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:31.788782 8078 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
W20260318 08:59:31.795754 7826 leader_election.cc:343] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185): Illegal state: must be running to vote when last-logged opid is not known
I20260318 08:59:31.799439 8067 tablet_bootstrap.cc:492] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836: Bootstrap starting.
I20260318 08:59:31.803470 8071 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:31.804098 8071 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.807392 8067 tablet_bootstrap.cc:654] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:31.808429 8078 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:31.810730 8065 tablet_bootstrap.cc:492] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0: No bootstrap required, opened a new log
I20260318 08:59:31.811283 8065 ts_tablet_manager.cc:1403] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.003s
I20260318 08:59:31.814139 8065 raft_consensus.cc:359] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.814883 8065 raft_consensus.cc:385] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:31.817109 8065 raft_consensus.cc:740] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06af5a588953450dbb0862d99ddf80f0, State: Initialized, Role: FOLLOWER
I20260318 08:59:31.817840 8065 consensus_queue.cc:260] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.819428 7862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 08:59:31.819768 8071 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:31.820426 7862 raft_consensus.cc:2468] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41e4ea383dc24ae5a6ef557c49dff836 in term 0.
I20260318 08:59:31.822104 7984 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836; no voters:
I20260318 08:59:31.825701 8078 raft_consensus.cc:2804] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260318 08:59:31.826771 8078 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 08:59:31.827280 8078 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:31.827751 8065 ts_tablet_manager.cc:1434] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20260318 08:59:31.836730 7791 catalog_manager.cc:1269] Loaded cluster ID: c582dd9307024b9483bfbf805eda4259
I20260318 08:59:31.837174 7791 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9: loading cluster ID for follower catalog manager: success
I20260318 08:59:31.852932 8069 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:31.853470 8069 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.879441 7804 catalog_manager.cc:1269] Loaded cluster ID: c582dd9307024b9483bfbf805eda4259
I20260318 08:59:31.879923 7804 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2: loading cluster ID for follower catalog manager: success
I20260318 08:59:31.899657 7940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 08:59:31.908354 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 08:59:31.910256 7939 raft_consensus.cc:2468] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06af5a588953450dbb0862d99ddf80f0 in term 0.
I20260318 08:59:31.918457 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:31.916821 8069 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:31.925443 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
W20260318 08:59:31.931710 7826 leader_election.cc:343] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185): Illegal state: must be running to vote when last-logged opid is not known
W20260318 08:59:31.936378 7981 leader_election.cc:343] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207): Illegal state: must be running to vote when last-logged opid is not known
W20260318 08:59:31.938220 7822 leader_election.cc:343] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207): Illegal state: must be running to vote when last-logged opid is not known
I20260318 08:59:31.938735 7822 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0; no voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836
I20260318 08:59:31.939754 8069 raft_consensus.cc:2749] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260318 08:59:31.938144 7791 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9: acquiring CA information for follower catalog manager: success
I20260318 08:59:31.940038 7822 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1; no voters: 41e4ea383dc24ae5a6ef557c49dff836
I20260318 08:59:31.944401 8069 raft_consensus.cc:2804] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260318 08:59:31.944818 8069 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 08:59:31.945171 8069 raft_consensus.cc:3060] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:31.949018 8066 tablet_bootstrap.cc:492] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1: No bootstrap required, opened a new log
I20260318 08:59:31.949607 8066 ts_tablet_manager.cc:1403] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1: Time spent bootstrapping tablet: real 0.213s user 0.031s sys 0.034s
I20260318 08:59:31.950335 8067 tablet_bootstrap.cc:492] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836: No bootstrap required, opened a new log
I20260318 08:59:31.950966 8067 ts_tablet_manager.cc:1403] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836: Time spent bootstrapping tablet: real 0.152s user 0.031s sys 0.049s
I20260318 08:59:31.951725 7791 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260318 08:59:31.958184 8066 raft_consensus.cc:359] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.958842 8066 raft_consensus.cc:385] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:31.959153 8066 raft_consensus.cc:740] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d02c582e0d940cdbd7b66d2c951c9e1, State: Initialized, Role: FOLLOWER
I20260318 08:59:31.959847 8066 consensus_queue.cc:260] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.961172 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:31.962335 8066 ts_tablet_manager.cc:1434] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260318 08:59:31.952020 7804 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2: acquiring CA information for follower catalog manager: success
I20260318 08:59:31.953346 8078 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.955092 8067 raft_consensus.cc:359] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.968601 7862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0"
I20260318 08:59:31.969349 7862 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:31.971105 8078 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 1 election: Requested vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:31.956418 8069 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.974454 8069 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:31.975521 8017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 08:59:31.978844 7804 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260318 08:59:31.979415 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1"
I20260318 08:59:31.988548 7939 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:31.988914 8066 tablet_bootstrap.cc:492] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1: Bootstrap starting.
I20260318 08:59:31.989565 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 08:59:31.990474 7861 raft_consensus.cc:2393] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in current term 1: Already voted for candidate 06af5a588953450dbb0862d99ddf80f0 in this term.
I20260318 08:59:31.979709 7940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1"
I20260318 08:59:31.991560 7940 raft_consensus.cc:3060] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:31.986513 7862 raft_consensus.cc:2468] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41e4ea383dc24ae5a6ef557c49dff836 in term 1.
I20260318 08:59:31.996340 8066 tablet_bootstrap.cc:654] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:31.996201 7984 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836; no voters:
I20260318 08:59:31.968581 8067 raft_consensus.cc:385] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:31.997329 8067 raft_consensus.cc:740] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e4ea383dc24ae5a6ef557c49dff836, State: Initialized, Role: FOLLOWER
I20260318 08:59:31.998265 8067 consensus_queue.cc:260] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:31.999269 8018 raft_consensus.cc:2468] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 0.
I20260318 08:59:32.003082 7939 raft_consensus.cc:2468] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41e4ea383dc24ae5a6ef557c49dff836 in term 1.
I20260318 08:59:32.003404 8078 raft_consensus.cc:2804] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 08:59:32.003566 7905 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0
I20260318 08:59:32.005748 8078 raft_consensus.cc:697] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 LEADER]: Becoming Leader. State: Replica: 41e4ea383dc24ae5a6ef557c49dff836, State: Running, Role: LEADER
I20260318 08:59:31.999233 7940 raft_consensus.cc:2468] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06af5a588953450dbb0862d99ddf80f0 in term 1.
I20260318 08:59:32.006786 8071 raft_consensus.cc:2764] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260318 08:59:32.007246 8066 tablet_bootstrap.cc:492] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1: No bootstrap required, opened a new log
I20260318 08:59:32.007879 8066 ts_tablet_manager.cc:1403] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20260318 08:59:32.007709 7822 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1; no voters: 41e4ea383dc24ae5a6ef557c49dff836
I20260318 08:59:32.006759 8078 consensus_queue.cc:237] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:32.010546 8069 raft_consensus.cc:2804] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 08:59:32.010783 8066 raft_consensus.cc:359] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:32.011621 8066 raft_consensus.cc:385] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:32.011931 8066 raft_consensus.cc:740] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d02c582e0d940cdbd7b66d2c951c9e1, State: Initialized, Role: FOLLOWER
I20260318 08:59:32.012602 8067 ts_tablet_manager.cc:1434] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836: Time spent starting tablet: real 0.061s user 0.010s sys 0.000s
I20260318 08:59:32.012743 8066 consensus_queue.cc:260] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:32.014271 8067 tablet_bootstrap.cc:492] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836: Bootstrap starting.
I20260318 08:59:32.023327 8066 ts_tablet_manager.cc:1434] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260318 08:59:32.040872 7597 catalog_manager.cc:5671] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67). New cstate: current_term: 1 leader_uuid: "41e4ea383dc24ae5a6ef557c49dff836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } health_report { overall_health: HEALTHY } } }
I20260318 08:59:32.047470 8067 tablet_bootstrap.cc:654] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836: Neither blocks nor log segments found. Creating new log.
I20260318 08:59:32.049330 8069 raft_consensus.cc:697] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 1 LEADER]: Becoming Leader. State: Replica: 06af5a588953450dbb0862d99ddf80f0, State: Running, Role: LEADER
I20260318 08:59:32.050886 8069 consensus_queue.cc:237] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:32.068089 8067 tablet_bootstrap.cc:492] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836: No bootstrap required, opened a new log
I20260318 08:59:32.081099 8067 ts_tablet_manager.cc:1403] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.011s
I20260318 08:59:32.084817 8067 raft_consensus.cc:359] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:32.085887 8067 raft_consensus.cc:385] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 08:59:32.086498 8067 raft_consensus.cc:740] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41e4ea383dc24ae5a6ef557c49dff836, State: Initialized, Role: FOLLOWER
I20260318 08:59:32.087368 8067 consensus_queue.cc:260] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:32.090065 8067 ts_tablet_manager.cc:1434] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260318 08:59:32.099527 8078 consensus_queue.cc:1048] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 08:59:32.100455 8071 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:32.100961 8071 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:32.102473 8071 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185), 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351)
I20260318 08:59:32.103909 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:32.104589 8018 raft_consensus.cc:2468] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 0.
I20260318 08:59:32.105504 7862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 08:59:32.105576 7905 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836; no voters:
I20260318 08:59:32.106343 7862 raft_consensus.cc:2468] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 0.
I20260318 08:59:32.106725 8093 raft_consensus.cc:2804] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260318 08:59:32.107182 8093 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 08:59:32.107545 8093 raft_consensus.cc:3060] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:32.117491 8093 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:32.120373 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 08:59:32.120999 8018 raft_consensus.cc:3060] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:32.122702 8096 consensus_queue.cc:1048] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260318 08:59:32.126837 8093 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 1 election: Requested vote from peers 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185), 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351)
I20260318 08:59:32.128782 8018 raft_consensus.cc:2468] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 1.
I20260318 08:59:32.130035 7905 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836; no voters:
I20260318 08:59:32.131098 8093 raft_consensus.cc:2804] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 08:59:32.131862 8093 raft_consensus.cc:697] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 3d02c582e0d940cdbd7b66d2c951c9e1, State: Running, Role: LEADER
I20260318 08:59:32.132879 8093 consensus_queue.cc:237] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:32.133340 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0"
I20260318 08:59:32.134186 7861 raft_consensus.cc:3060] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:32.162894 8018 raft_consensus.cc:3060] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 0 FOLLOWER]: Advancing to term 1
I20260318 08:59:32.173990 8069 consensus_queue.cc:1048] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 08:59:32.185568 7861 raft_consensus.cc:2468] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 1.
I20260318 08:59:32.187501 8069 consensus_queue.cc:1048] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 08:59:32.253244 7596 catalog_manager.cc:5671] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65). New cstate: current_term: 1 leader_uuid: "06af5a588953450dbb0862d99ddf80f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } health_report { overall_health: UNKNOWN } } }
I20260318 08:59:32.299166 8104 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e45b189483e74bc089b2963727d99490)
I20260318 08:59:32.300051 8104 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:32.303475 8104 leader_election.cc:290] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e45b189483e74bc089b2963727d99490 (127.6.2.126:40111), 07f47abe644e4c6786bd63c1251a0ac2 (127.6.2.125:34659)
I20260318 08:59:32.311398 7607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "e45b189483e74bc089b2963727d99490" is_pre_election: true
I20260318 08:59:32.352020 8106 raft_consensus.cc:493] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e45b189483e74bc089b2963727d99490)
I20260318 08:59:32.352639 8106 raft_consensus.cc:515] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:32.355043 8106 leader_election.cc:290] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e45b189483e74bc089b2963727d99490 (127.6.2.126:40111), 1e52aab3abc24498a28a69e7c89452f9 (127.6.2.124:42683)
I20260318 08:59:32.356029 7607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07f47abe644e4c6786bd63c1251a0ac2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "e45b189483e74bc089b2963727d99490" is_pre_election: true
I20260318 08:59:32.432888 8103 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:32.433499 8103 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:32.437026 8103 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:32.437656 7940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 08:59:32.453219 8093 consensus_queue.cc:1048] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 08:59:32.568657 8077 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:32.569365 8077 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:32.574704 8077 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:32.575819 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 08:59:32.678534 7672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "07f47abe644e4c6786bd63c1251a0ac2" is_pre_election: true
I20260318 08:59:32.678529 7737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07f47abe644e4c6786bd63c1251a0ac2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "1e52aab3abc24498a28a69e7c89452f9" is_pre_election: true
I20260318 08:59:32.679415 7672 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e52aab3abc24498a28a69e7c89452f9 in term 1.
I20260318 08:59:32.679530 7737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07f47abe644e4c6786bd63c1251a0ac2 in term 1.
I20260318 08:59:32.681100 7645 leader_election.cc:304] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07f47abe644e4c6786bd63c1251a0ac2, 1e52aab3abc24498a28a69e7c89452f9; no voters: e45b189483e74bc089b2963727d99490
I20260318 08:59:32.694569 7671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "07f47abe644e4c6786bd63c1251a0ac2" is_pre_election: true
I20260318 08:59:32.695226 7671 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e52aab3abc24498a28a69e7c89452f9 in term 1.
I20260318 08:59:32.696300 7713 leader_election.cc:304] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07f47abe644e4c6786bd63c1251a0ac2, 1e52aab3abc24498a28a69e7c89452f9; no voters: e45b189483e74bc089b2963727d99490
I20260318 08:59:32.740439 8104 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260318 08:59:32.740934 8104 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e45b189483e74bc089b2963727d99490)
I20260318 08:59:32.741335 8104 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:32.748307 8104 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:32.753103 7671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "07f47abe644e4c6786bd63c1251a0ac2"
I20260318 08:59:32.755654 8104 leader_election.cc:290] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [CANDIDATE]: Term 2 election: Requested vote from peers e45b189483e74bc089b2963727d99490 (127.6.2.126:40111), 07f47abe644e4c6786bd63c1251a0ac2 (127.6.2.125:34659)
I20260318 08:59:32.771072 7607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "e45b189483e74bc089b2963727d99490"
I20260318 08:59:32.772686 7712 leader_election.cc:304] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e52aab3abc24498a28a69e7c89452f9; no voters: 07f47abe644e4c6786bd63c1251a0ac2, e45b189483e74bc089b2963727d99490
I20260318 08:59:32.773428 8104 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260318 08:59:32.775081 8093 consensus_queue.cc:1048] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 08:59:32.777598 7737 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 2 FOLLOWER]: Rejecting Update request from peer e45b189483e74bc089b2963727d99490 for earlier term 1. Current term is 2. Ops: []
I20260318 08:59:32.780187 8111 consensus_queue.cc:1059] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 }, Status: INVALID_TERM, Last received: 1.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260318 08:59:32.782714 8106 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260318 08:59:32.783140 8106 raft_consensus.cc:493] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e45b189483e74bc089b2963727d99490)
I20260318 08:59:32.783574 8106 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:32.789871 8069 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 41e4ea383dc24ae5a6ef557c49dff836)
I20260318 08:59:32.790441 8069 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:32.792376 8069 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:32.792483 8106 raft_consensus.cc:515] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:32.793428 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 08:59:32.794137 7862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 08:59:32.795759 7737 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 2 FOLLOWER]: Rejecting Update request from peer e45b189483e74bc089b2963727d99490 for earlier term 1. Current term is 2. Ops: []
I20260318 08:59:32.796056 7984 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1
I20260318 08:59:32.796789 7607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07f47abe644e4c6786bd63c1251a0ac2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "e45b189483e74bc089b2963727d99490"
I20260318 08:59:32.797093 8077 raft_consensus.cc:2749] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260318 08:59:32.798381 8111 consensus_queue.cc:1059] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 }, Status: INVALID_TERM, Last received: 1.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260318 08:59:32.799520 8106 leader_election.cc:290] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [CANDIDATE]: Term 2 election: Requested vote from peers e45b189483e74bc089b2963727d99490 (127.6.2.126:40111), 1e52aab3abc24498a28a69e7c89452f9 (127.6.2.124:42683)
I20260318 08:59:32.800060 7737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07f47abe644e4c6786bd63c1251a0ac2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "1e52aab3abc24498a28a69e7c89452f9"
I20260318 08:59:32.802408 7737 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 07f47abe644e4c6786bd63c1251a0ac2 in current term 2: Already voted for candidate 1e52aab3abc24498a28a69e7c89452f9 in this term.
I20260318 08:59:32.803375 7645 leader_election.cc:304] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07f47abe644e4c6786bd63c1251a0ac2; no voters: 1e52aab3abc24498a28a69e7c89452f9, e45b189483e74bc089b2963727d99490
I20260318 08:59:32.804073 8106 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260318 08:59:32.805033 7805 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.050s sys 0.036s
W20260318 08:59:32.805568 7805 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.614s user 0.050s sys 0.036s
I20260318 08:59:32.810261 7805 raft_consensus.cc:3055] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 1 LEADER]: Stepping down as leader of term 1
I20260318 08:59:32.810626 7805 raft_consensus.cc:740] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e45b189483e74bc089b2963727d99490, State: Running, Role: LEADER
I20260318 08:59:32.811614 7805 consensus_queue.cc:260] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:32.816291 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:32.817090 8016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:32.819366 7826 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0; no voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836
I20260318 08:59:32.821079 7826 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0; no voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836
I20260318 08:59:32.822175 8069 raft_consensus.cc:2749] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260318 08:59:32.822722 8103 raft_consensus.cc:2749] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260318 08:59:32.829741 7737 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 2 FOLLOWER]: Rejecting Update request from peer e45b189483e74bc089b2963727d99490 for earlier term 1. Current term is 2. Ops: [1.8-1.8]
I20260318 08:59:32.842283 7805 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 1 FOLLOWER]: Advancing to term 2
W20260318 08:59:32.932605 8053 master_proxy_rpc.cc:203] Re-attempting IsCreateTableDone request to leader Master (127.6.2.126:40111)
I20260318 08:59:32.966143 7805 raft_consensus.cc:493] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:32.966701 7805 raft_consensus.cc:515] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:32.968781 7805 leader_election.cc:290] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 07f47abe644e4c6786bd63c1251a0ac2 (127.6.2.125:34659), 1e52aab3abc24498a28a69e7c89452f9 (127.6.2.124:42683)
I20260318 08:59:32.970343 7671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e45b189483e74bc089b2963727d99490" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "07f47abe644e4c6786bd63c1251a0ac2" is_pre_election: true
I20260318 08:59:32.970996 7671 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e45b189483e74bc089b2963727d99490 in term 2.
I20260318 08:59:32.973241 7584 leader_election.cc:304] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f47abe644e4c6786bd63c1251a0ac2, e45b189483e74bc089b2963727d99490; no voters:
I20260318 08:59:32.974318 7805 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260318 08:59:32.973385 7737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e45b189483e74bc089b2963727d99490" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1e52aab3abc24498a28a69e7c89452f9" is_pre_election: true
I20260318 08:59:32.974709 7805 raft_consensus.cc:493] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 08:59:32.975097 7805 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 2 FOLLOWER]: Advancing to term 3
I20260318 08:59:32.975175 7737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e45b189483e74bc089b2963727d99490 in term 2.
I20260318 08:59:32.980365 7805 raft_consensus.cc:515] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:32.983161 7671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e45b189483e74bc089b2963727d99490" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "07f47abe644e4c6786bd63c1251a0ac2"
I20260318 08:59:32.983798 7671 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 2 FOLLOWER]: Advancing to term 3
I20260318 08:59:32.984370 7805 leader_election.cc:290] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [CANDIDATE]: Term 3 election: Requested vote from peers 07f47abe644e4c6786bd63c1251a0ac2 (127.6.2.125:34659), 1e52aab3abc24498a28a69e7c89452f9 (127.6.2.124:42683)
I20260318 08:59:32.985035 7737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e45b189483e74bc089b2963727d99490" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1e52aab3abc24498a28a69e7c89452f9"
I20260318 08:59:32.985716 7737 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 2 FOLLOWER]: Advancing to term 3
I20260318 08:59:32.990836 7671 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e45b189483e74bc089b2963727d99490 in term 3.
I20260318 08:59:32.992885 7584 leader_election.cc:304] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f47abe644e4c6786bd63c1251a0ac2, e45b189483e74bc089b2963727d99490; no voters:
I20260318 08:59:32.993399 7737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e45b189483e74bc089b2963727d99490 in term 3.
I20260318 08:59:32.993814 7805 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 3 FOLLOWER]: Leader election won for term 3
I20260318 08:59:32.994632 7805 raft_consensus.cc:697] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 3 LEADER]: Becoming Leader. State: Replica: e45b189483e74bc089b2963727d99490, State: Running, Role: LEADER
I20260318 08:59:32.995554 7805 consensus_queue.cc:237] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:33.000204 8058 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e45b189483e74bc089b2963727d99490. Latest consensus state: current_term: 3 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:33.000821 8058 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:33.088496 7737 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Refusing update from remote peer e45b189483e74bc089b2963727d99490: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20260318 08:59:33.089823 8058 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260318 08:59:33.096927 8104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e45b189483e74bc089b2963727d99490. Latest consensus state: current_term: 3 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:33.097726 8104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:33.104694 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:33.105556 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:33.105904 7596 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35744 (request call id 15) took 1039 ms. Trace:
I20260318 08:59:33.106060 7671 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 3 FOLLOWER]: Refusing update from remote peer e45b189483e74bc089b2963727d99490: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20260318 08:59:33.108629 7805 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260318 08:59:33.109593 8104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:33.110375 8104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:33.106395 7596 rpcz_store.cc:276] 0318 08:59:32.066314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0318 08:59:32.066920 (+ 606us) service_pool.cc:225] Handling call
0318 08:59:33.105841 (+1038921us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0318 08:59:32.276328 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0318 08:59:32.280950 (+ 4622us) write_op.cc:432] Acquiring schema lock in shared mode
0318 08:59:32.280991 (+ 41us) write_op.cc:435] Acquired schema lock
0318 08:59:32.281007 (+ 16us) tablet.cc:661] Decoding operations
0318 08:59:32.281386 (+ 379us) write_op.cc:620] Acquiring the partition lock for write op
0318 08:59:32.281478 (+ 92us) write_op.cc:641] Partition lock acquired for write op
0318 08:59:32.281524 (+ 46us) tablet.cc:684] Acquiring locks for 1 operations
0318 08:59:32.281745 (+ 221us) tablet.cc:700] Row locks acquired
0318 08:59:32.281761 (+ 16us) write_op.cc:260] PREPARE: finished
0318 08:59:32.281928 (+ 167us) write_op.cc:270] Start()
0318 08:59:32.282055 (+ 127us) write_op.cc:276] Timestamp: P: 1773824372281875 usec, L: 0
0318 08:59:32.282071 (+ 16us) op_driver.cc:348] REPLICATION: starting
0318 08:59:32.807621 (+525550us) log.cc:844] Serialized 534 byte log entry
0318 08:59:32.810860 (+ 3239us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54000e06d4. stack: 00007fbbbb1e0423 00007fbbbb1e0320 00007fbbbab32574 00000000003bff74 00000000003bff09 00007fbbc1b8e7a1 00007fbbc1c20d58 00007fbbc20927b1 00007fbbc2091f88 00007fbbc2096c21 00007fbbc2096bd9 00007fbbc2096b69 00007fbbc2096b31 00007fbbc2095d5d 00000000003f8f34 00000000003f8d79
0318 08:59:32.826987 (+ 16127us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b54000e0c20. stack: 00007fbbbb1e0423 00007fbbbb1e0320 00007fbbbab32574 00000000003bff74 00000000003bff09 00007fbbc1b8e7a1 00007fbbc1ba5d56 00007fbbc1b9f6ca 00007fbbc1b9e944 00007fbbc1b7f0da 00007fbbc1b85f83 00007fbbc1b85f09 00007fbbc1b85e99 00007fbbc1b85e61 00007fbbc1b8502d 00000000003f8f34
0318 08:59:33.102413 (+275426us) op_driver.cc:464] REPLICATION: finished
0318 08:59:33.103686 (+ 1273us) write_op.cc:301] APPLY: starting
0318 08:59:33.103762 (+ 76us) tablet.cc:1366] starting BulkCheckPresence
0318 08:59:33.103854 (+ 92us) tablet.cc:1369] finished BulkCheckPresence
0318 08:59:33.103875 (+ 21us) tablet.cc:1371] starting ApplyRowOperation cycle
0318 08:59:33.104072 (+ 197us) tablet.cc:1382] finished ApplyRowOperation cycle
0318 08:59:33.104225 (+ 153us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0318 08:59:33.104269 (+ 44us) write_op.cc:312] APPLY: finished
0318 08:59:33.104441 (+ 172us) log.cc:844] Serialized 22 byte log entry
0318 08:59:33.104603 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0318 08:59:33.104679 (+ 76us) write_op.cc:454] Released schema lock
0318 08:59:33.104718 (+ 39us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":1077,"apply.run_cpu_time_us":1310,"apply.run_wall_time_us":1311,"mutex_wait_us":1090,"num_ops":1,"prepare.queue_time_us":2634,"prepare.run_cpu_time_us":12665,"prepare.run_wall_time_us":543283,"raft.queue_time_us":1359,"raft.run_cpu_time_us":3204,"raft.run_wall_time_us":17050,"replication_time_us":820193,"spinlock_wait_cycles":1210038272,"thread_start_us":754,"threads_started":1}]]}
I20260318 08:59:33.113517 8118 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 08:59:33.122450 8118 catalog_manager.cc:679] Loaded metadata for table test-table [id=81693cd942f64504990e0adaa14b87f5]
I20260318 08:59:33.126217 8124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:33.127063 8124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:33.127172 8106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e45b189483e74bc089b2963727d99490. Latest consensus state: current_term: 3 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:33.127993 8106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 08:59:33.130378 8118 tablet_loader.cc:96] loaded metadata for tablet 0f3811d2d92f41c2b7301baa5ab3f1cb (table test-table [id=81693cd942f64504990e0adaa14b87f5])
I20260318 08:59:33.131927 8118 tablet_loader.cc:96] loaded metadata for tablet 5adac9d378be4afdbe6723195ec9bd11 (table test-table [id=81693cd942f64504990e0adaa14b87f5])
I20260318 08:59:33.133332 8118 tablet_loader.cc:96] loaded metadata for tablet a936692bfd8f4eaa9b58cb7cf64d05d4 (table test-table [id=81693cd942f64504990e0adaa14b87f5])
I20260318 08:59:33.134857 8118 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 08:59:33.138139 8118 catalog_manager.cc:1269] Loaded cluster ID: c582dd9307024b9483bfbf805eda4259
I20260318 08:59:33.138386 8118 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 08:59:33.141664 8118 catalog_manager.cc:1514] Loading token signing keys...
I20260318 08:59:33.144606 8118 catalog_manager.cc:6055] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490: Loaded TSK: 0
I20260318 08:59:33.145605 8118 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 08:59:33.214552 8111 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e45b189483e74bc089b2963727d99490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 08:59:33.215420 8111 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: This master's current role is: LEADER
I20260318 08:59:33.281338 8045 heartbeater.cc:499] Master 127.6.2.126:40111 was elected leader, sending a full tablet report...
I20260318 08:59:33.287923 7594 catalog_manager.cc:5671] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66). New cstate: current_term: 1 leader_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } }
W20260318 08:59:33.298024 7628 debug-util.cc:398] Leaking SignalData structure 0x7b0800049360 after lost signal to thread 6156
W20260318 08:59:33.299122 7628 debug-util.cc:398] Leaking SignalData structure 0x7b0800133c40 after lost signal to thread 7631
W20260318 08:59:33.300084 7628 debug-util.cc:398] Leaking SignalData structure 0x7b08000370e0 after lost signal to thread 7696
W20260318 08:59:33.300990 7628 debug-util.cc:398] Leaking SignalData structure 0x7b080024e960 after lost signal to thread 7761
I20260318 08:59:33.302333 7968 heartbeater.cc:499] Master 127.6.2.126:40111 was elected leader, sending a full tablet report...
W20260318 08:59:33.305891 7628 debug-util.cc:398] Leaking SignalData structure 0x7b080024fa20 after lost signal to thread 7886
W20260318 08:59:33.306950 7628 debug-util.cc:398] Leaking SignalData structure 0x7b08002501e0 after lost signal to thread 7964
W20260318 08:59:33.308019 7628 debug-util.cc:398] Leaking SignalData structure 0x7b0800247500 after lost signal to thread 8042
I20260318 08:59:33.324450 7889 heartbeater.cc:499] Master 127.6.2.126:40111 was elected leader, sending a full tablet report...
I20260318 08:59:33.632236 8126 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e45b189483e74bc089b2963727d99490)
I20260318 08:59:33.632838 8126 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:33.644757 7607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "e45b189483e74bc089b2963727d99490" is_pre_election: true
I20260318 08:59:33.648595 7671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "07f47abe644e4c6786bd63c1251a0ac2" is_pre_election: true
I20260318 08:59:33.650240 7713 leader_election.cc:304] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e52aab3abc24498a28a69e7c89452f9; no voters: 07f47abe644e4c6786bd63c1251a0ac2, e45b189483e74bc089b2963727d99490
I20260318 08:59:33.656113 8126 leader_election.cc:290] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e45b189483e74bc089b2963727d99490 (127.6.2.126:40111), 07f47abe644e4c6786bd63c1251a0ac2 (127.6.2.125:34659)
I20260318 08:59:33.656762 8126 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260318 08:59:33.676055 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800035f20 after lost signal to thread 6156
W20260318 08:59:33.677121 7883 debug-util.cc:398] Leaking SignalData structure 0x7b080017ef40 after lost signal to thread 7631
W20260318 08:59:33.678021 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800192600 after lost signal to thread 7696
W20260318 08:59:33.678809 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800183060 after lost signal to thread 7761
W20260318 08:59:33.680200 7883 debug-util.cc:398] Leaking SignalData structure 0x7b080025b940 after lost signal to thread 7886
W20260318 08:59:33.680505 7883 debug-util.cc:398] Leaking SignalData structure 0x7b080025b9a0 after lost signal to thread 7889
W20260318 08:59:33.681447 7883 debug-util.cc:398] Leaking SignalData structure 0x7b08000fd7a0 after lost signal to thread 7964
W20260318 08:59:33.682219 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800220680 after lost signal to thread 8042
W20260318 08:59:33.682483 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800220380 after lost signal to thread 8044
W20260318 08:59:33.682682 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800220360 after lost signal to thread 8045
W20260318 08:59:34.370786 8144 log.cc:927] Time spent T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260318 08:59:47.050953 8183 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 08:59:47.052434 8183 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.057461 7862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 08:59:47.057718 8016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:47.079317 8183 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:47.340188 8187 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d02c582e0d940cdbd7b66d2c951c9e1)
I20260318 08:59:47.340956 8187 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 08:59:47.347442 8192 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 41e4ea383dc24ae5a6ef557c49dff836)
I20260318 08:59:47.348779 8192 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.351009 8187 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:47.369728 8183 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 41e4ea383dc24ae5a6ef557c49dff836)
I20260318 08:59:47.373337 8183 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.375161 8192 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:47.377987 8183 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:47.379632 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 08:59:47.380540 7861 raft_consensus.cc:2468] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 1.
I20260318 08:59:47.383517 7906 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1; no voters:
I20260318 08:59:47.386170 8188 raft_consensus.cc:2804] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260318 08:59:47.386385 8194 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e45b189483e74bc089b2963727d99490)
I20260318 08:59:47.386619 8188 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 41e4ea383dc24ae5a6ef557c49dff836)
I20260318 08:59:47.387064 8194 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:47.387072 8188 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:47.393682 7607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "e45b189483e74bc089b2963727d99490" is_pre_election: true
W20260318 08:59:47.379771 7905 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.6.2.67:45185 timed out after 0.272s (SENT) [suppressed 6 similar messages]
I20260318 08:59:47.398309 7671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "07f47abe644e4c6786bd63c1251a0ac2" is_pre_election: true
I20260318 08:59:47.398561 8193 raft_consensus.cc:493] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e45b189483e74bc089b2963727d99490)
I20260318 08:59:47.398703 8016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:47.399199 8193 raft_consensus.cc:515] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 08:59:47.400501 7713 leader_election.cc:304] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e52aab3abc24498a28a69e7c89452f9; no voters: 07f47abe644e4c6786bd63c1251a0ac2, e45b189483e74bc089b2963727d99490
I20260318 08:59:47.401567 8194 leader_election.cc:290] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e45b189483e74bc089b2963727d99490 (127.6.2.126:40111), 07f47abe644e4c6786bd63c1251a0ac2 (127.6.2.125:34659)
I20260318 08:59:47.402196 8194 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260318 08:59:47.405431 8182 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 08:59:47.405943 8182 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.406266 7607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07f47abe644e4c6786bd63c1251a0ac2" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "e45b189483e74bc089b2963727d99490" is_pre_election: true
I20260318 08:59:47.411664 7940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 08:59:47.414712 7862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 08:59:47.416172 8193 leader_election.cc:290] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e45b189483e74bc089b2963727d99490 (127.6.2.126:40111), 1e52aab3abc24498a28a69e7c89452f9 (127.6.2.124:42683)
I20260318 08:59:47.418561 7938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 08:59:47.421537 8017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:47.422264 8017 raft_consensus.cc:2468] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 1.
I20260318 08:59:47.399194 8188 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.423573 7938 raft_consensus.cc:2393] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06af5a588953450dbb0862d99ddf80f0 in current term 2: Already voted for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in this term.
I20260318 08:59:47.427218 8188 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 2 election: Requested vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:47.427357 7939 raft_consensus.cc:1240] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Rejecting Update request from peer 41e4ea383dc24ae5a6ef557c49dff836 for earlier term 1. Current term is 2. Ops: []
I20260318 08:59:47.427654 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0"
I20260318 08:59:47.436184 7737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07f47abe644e4c6786bd63c1251a0ac2" candidate_term: 4 candidate_status { last_received { term: 3 index: 10 } } ignore_live_leader: false dest_uuid: "1e52aab3abc24498a28a69e7c89452f9" is_pre_election: true
I20260318 08:59:47.437007 8196 consensus_queue.cc:1059] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260318 08:59:47.437644 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 08:59:47.438196 8196 raft_consensus.cc:3055] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 LEADER]: Stepping down as leader of term 1
I20260318 08:59:47.438551 8196 raft_consensus.cc:740] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 41e4ea383dc24ae5a6ef557c49dff836, State: Running, Role: LEADER
I20260318 08:59:47.439371 8182 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:47.440524 7645 leader_election.cc:304] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 07f47abe644e4c6786bd63c1251a0ac2; no voters: 1e52aab3abc24498a28a69e7c89452f9, e45b189483e74bc089b2963727d99490
I20260318 08:59:47.440649 7905 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0
I20260318 08:59:47.439441 8196 consensus_queue.cc:260] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.441838 8193 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260318 08:59:47.441890 8183 raft_consensus.cc:2804] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260318 08:59:47.442631 8183 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 08:59:47.442879 8196 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:47.443030 8183 raft_consensus.cc:3060] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:47.442996 7905 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1; no voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836
I20260318 08:59:47.445751 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 08:59:47.446785 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:47.447242 8016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:47.451136 8183 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.452044 7826 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0; no voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836
I20260318 08:59:47.452951 8179 raft_consensus.cc:2749] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260318 08:59:47.453300 8183 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 2 election: Requested vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:47.445686 8188 raft_consensus.cc:2749] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260318 08:59:47.454739 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0"
I20260318 08:59:47.458216 8018 raft_consensus.cc:2468] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06af5a588953450dbb0862d99ddf80f0 in term 2.
I20260318 08:59:47.458562 7981 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1
I20260318 08:59:47.459364 8196 raft_consensus.cc:2749] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260318 08:59:47.460284 7826 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836; no voters: 3d02c582e0d940cdbd7b66d2c951c9e1
I20260318 08:59:47.460335 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 08:59:47.461236 8179 raft_consensus.cc:2804] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260318 08:59:47.461665 8179 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 41e4ea383dc24ae5a6ef557c49dff836)
I20260318 08:59:47.461930 7905 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1; no voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836
I20260318 08:59:47.462036 8179 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:47.462687 8188 raft_consensus.cc:2749] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260318 08:59:47.471487 8179 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.474601 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1"
I20260318 08:59:47.475250 8179 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:47.475421 7939 raft_consensus.cc:2393] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 06af5a588953450dbb0862d99ddf80f0 in current term 2: Already voted for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in this term.
I20260318 08:59:47.476091 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 08:59:47.482122 8018 raft_consensus.cc:2468] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06af5a588953450dbb0862d99ddf80f0 in term 2.
I20260318 08:59:47.483048 7826 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836; no voters: 3d02c582e0d940cdbd7b66d2c951c9e1
I20260318 08:59:47.483860 8179 raft_consensus.cc:2804] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260318 08:59:47.484380 8179 raft_consensus.cc:697] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 2 LEADER]: Becoming Leader. State: Replica: 06af5a588953450dbb0862d99ddf80f0, State: Running, Role: LEADER
I20260318 08:59:47.485332 8179 consensus_queue.cc:237] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.494542 7589 catalog_manager.cc:5671] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 reported cstate change: term changed from 1 to 2, leader changed from 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67) to 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65). New cstate: current_term: 2 leader_uuid: "06af5a588953450dbb0862d99ddf80f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } health_report { overall_health: UNKNOWN } } }
I20260318 08:59:47.503458 7939 raft_consensus.cc:1240] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Rejecting Update request from peer 06af5a588953450dbb0862d99ddf80f0 for earlier term 1. Current term is 2. Ops: []
I20260318 08:59:47.506057 8179 consensus_queue.cc:1059] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 08:59:47.507385 8191 raft_consensus.cc:3055] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 1 LEADER]: Stepping down as leader of term 1
I20260318 08:59:47.507843 8191 raft_consensus.cc:740] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 06af5a588953450dbb0862d99ddf80f0, State: Running, Role: LEADER
I20260318 08:59:47.508792 8191 consensus_queue.cc:260] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.510355 8191 raft_consensus.cc:3060] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:47.581321 7939 raft_consensus.cc:1275] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Refusing update from remote peer 06af5a588953450dbb0862d99ddf80f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260318 08:59:47.582732 8191 consensus_queue.cc:1048] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 08:59:47.599655 8018 raft_consensus.cc:1275] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Refusing update from remote peer 06af5a588953450dbb0862d99ddf80f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260318 08:59:47.601996 8191 consensus_queue.cc:1048] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 08:59:47.697980 8183 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:47.698514 8183 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.700289 8183 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:47.701140 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 08:59:47.701534 8196 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 08:59:47.701833 7861 raft_consensus.cc:2468] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 2.
I20260318 08:59:47.702147 8196 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.703043 7906 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1; no voters:
I20260318 08:59:47.703852 8183 raft_consensus.cc:2804] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260318 08:59:47.704219 8183 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 08:59:47.704587 8183 raft_consensus.cc:3060] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Advancing to term 3
I20260318 08:59:47.701565 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:47.705395 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 08:59:47.705506 8191 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 08:59:47.705636 8018 raft_consensus.cc:2468] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 1.
I20260318 08:59:47.706090 8196 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:47.706102 8191 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.707101 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 08:59:47.707497 7861 raft_consensus.cc:2468] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41e4ea383dc24ae5a6ef557c49dff836 in term 2.
I20260318 08:59:47.711244 8191 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:47.711869 7984 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836; no voters:
I20260318 08:59:47.713934 8016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 08:59:47.714538 8016 raft_consensus.cc:2468] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06af5a588953450dbb0862d99ddf80f0 in term 1.
I20260318 08:59:47.715297 8196 raft_consensus.cc:2804] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260318 08:59:47.715667 8196 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 08:59:47.716035 8196 raft_consensus.cc:3060] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Advancing to term 2
I20260318 08:59:47.716305 7938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 08:59:47.716673 7826 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836; no voters:
I20260318 08:59:47.718695 8183 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.723238 7939 raft_consensus.cc:2368] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41e4ea383dc24ae5a6ef557c49dff836 for earlier term 2. Current term is 3.
I20260318 08:59:47.723837 8191 raft_consensus.cc:2804] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260318 08:59:47.724267 8191 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 08:59:47.724639 8191 raft_consensus.cc:3060] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Advancing to term 3
I20260318 08:59:47.724651 8196 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.727532 7981 leader_election.cc:400] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207) with higher term. Message: Invalid argument: T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41e4ea383dc24ae5a6ef557c49dff836 for earlier term 2. Current term is 3.
I20260318 08:59:47.728401 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0"
I20260318 08:59:47.728816 8183 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 election: Requested vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:47.729056 8196 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 election: Requested vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 08:59:47.729995 7862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0"
I20260318 08:59:47.729216 8018 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 08:59:47.730633 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1"
I20260318 08:59:47.730635 8018 raft_consensus.cc:3060] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Advancing to term 3
I20260318 08:59:47.731410 7939 raft_consensus.cc:2368] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 41e4ea383dc24ae5a6ef557c49dff836 for earlier term 2. Current term is 3.
I20260318 08:59:47.732438 7981 leader_election.cc:400] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 election: Vote denied by peer 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207) with higher term. Message: Invalid argument: T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 41e4ea383dc24ae5a6ef557c49dff836 for earlier term 2. Current term is 3.
I20260318 08:59:47.732735 7981 leader_election.cc:403] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260318 08:59:47.734056 8191 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.735272 7861 raft_consensus.cc:2393] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in current term 3: Already voted for candidate 06af5a588953450dbb0862d99ddf80f0 in this term.
I20260318 08:59:47.736076 8191 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 08:59:47.736930 8016 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 08:59:47.737272 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1"
I20260318 08:59:47.737957 7939 raft_consensus.cc:2393] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 06af5a588953450dbb0862d99ddf80f0 in current term 3: Already voted for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in this term.
I20260318 08:59:47.738072 8018 raft_consensus.cc:2468] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 3.
I20260318 08:59:47.738613 8196 raft_consensus.cc:2749] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207) with higher term. Message: Invalid argument: T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 41e4ea383dc24ae5a6ef557c49dff836 for earlier term 2. Current term is 3.
I20260318 08:59:47.739003 7822 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0; no voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836
I20260318 08:59:47.739233 7905 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0
I20260318 08:59:47.739840 8191 raft_consensus.cc:2749] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260318 08:59:47.739953 8183 raft_consensus.cc:2804] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Leader election won for term 3
I20260318 08:59:47.740643 8183 raft_consensus.cc:697] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 LEADER]: Becoming Leader. State: Replica: 3d02c582e0d940cdbd7b66d2c951c9e1, State: Running, Role: LEADER
I20260318 08:59:47.741597 8183 consensus_queue.cc:237] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 08:59:47.749347 7593 catalog_manager.cc:5671] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 reported cstate change: term changed from 1 to 3, leader changed from 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65) to 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66). New cstate: current_term: 3 leader_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } health_report { overall_health: UNKNOWN } } }
I20260318 08:59:47.868798 7861 raft_consensus.cc:1275] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Refusing update from remote peer 3d02c582e0d940cdbd7b66d2c951c9e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260318 08:59:47.869961 8183 consensus_queue.cc:1048] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 08:59:47.870440 8018 raft_consensus.cc:1275] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Refusing update from remote peer 3d02c582e0d940cdbd7b66d2c951c9e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260318 08:59:47.871794 8183 consensus_queue.cc:1048] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260318 08:59:48.104761 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800220260 after lost signal to thread 6156
W20260318 08:59:48.105835 7883 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1340 after lost signal to thread 7631
W20260318 08:59:48.106711 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800153540 after lost signal to thread 7696
W20260318 08:59:48.107647 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800035fc0 after lost signal to thread 7761
W20260318 08:59:48.108989 7883 debug-util.cc:398] Leaking SignalData structure 0x7b080024f840 after lost signal to thread 7886
W20260318 08:59:48.109972 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800141ec0 after lost signal to thread 7964
W20260318 08:59:48.110941 7883 debug-util.cc:398] Leaking SignalData structure 0x7b0800056340 after lost signal to thread 8042
I20260318 09:00:02.778966 8236 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d02c582e0d940cdbd7b66d2c951c9e1)
I20260318 09:00:02.780361 8236 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 09:00:02.784880 8236 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 09:00:02.796350 8238 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e45b189483e74bc089b2963727d99490)
I20260318 09:00:02.796990 8238 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 09:00:02.800228 8238 leader_election.cc:290] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e45b189483e74bc089b2963727d99490 (127.6.2.126:40111), 07f47abe644e4c6786bd63c1251a0ac2 (127.6.2.125:34659)
I20260318 09:00:02.807308 8239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e45b189483e74bc089b2963727d99490)
I20260318 09:00:02.807828 8239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 09:00:02.810622 8239 leader_election.cc:290] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e45b189483e74bc089b2963727d99490 (127.6.2.126:40111), 1e52aab3abc24498a28a69e7c89452f9 (127.6.2.124:42683)
I20260318 09:00:02.818113 8231 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 09:00:02.818254 8240 raft_consensus.cc:493] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d02c582e0d940cdbd7b66d2c951c9e1)
I20260318 09:00:02.819006 8240 raft_consensus.cc:515] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 09:00:02.819222 8231 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 09:00:02.824266 8231 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 09:00:02.824805 8240 leader_election.cc:290] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 09:00:02.827944 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 09:00:02.828974 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5adac9d378be4afdbe6723195ec9bd11" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 09:00:02.829707 7606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07f47abe644e4c6786bd63c1251a0ac2" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "e45b189483e74bc089b2963727d99490" is_pre_election: true
I20260318 09:00:02.830156 7607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "e45b189483e74bc089b2963727d99490" is_pre_election: true
I20260318 09:00:02.830411 7981 leader_election.cc:304] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1
I20260318 09:00:02.831621 7737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07f47abe644e4c6786bd63c1251a0ac2" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "1e52aab3abc24498a28a69e7c89452f9" is_pre_election: true
I20260318 09:00:02.831777 8240 raft_consensus.cc:2749] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260318 09:00:02.832204 7737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07f47abe644e4c6786bd63c1251a0ac2 in term 3.
I20260318 09:00:02.834261 7645 leader_election.cc:304] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07f47abe644e4c6786bd63c1251a0ac2, 1e52aab3abc24498a28a69e7c89452f9; no voters: e45b189483e74bc089b2963727d99490
I20260318 09:00:02.835122 8239 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260318 09:00:02.835564 8239 raft_consensus.cc:493] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e45b189483e74bc089b2963727d99490)
I20260318 09:00:02.836000 8239 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 3 FOLLOWER]: Advancing to term 4
I20260318 09:00:02.837060 8240 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 09:00:02.837513 8240 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 09:00:02.840271 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 09:00:02.840442 8240 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 09:00:02.840617 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 09:00:02.841238 8240 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d02c582e0d940cdbd7b66d2c951c9e1)
I20260318 09:00:02.841940 7981 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1
I20260318 09:00:02.851230 8240 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 09:00:02.852380 8239 raft_consensus.cc:515] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 09:00:02.854008 7671 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 4 FOLLOWER]: Rejecting Update request from peer e45b189483e74bc089b2963727d99490 for earlier term 3. Current term is 4. Ops: []
I20260318 09:00:02.854959 7939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 09:00:02.855139 8240 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351)
I20260318 09:00:02.856034 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 09:00:02.855713 8240 raft_consensus.cc:2749] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260318 09:00:02.857440 7984 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1
I20260318 09:00:02.858115 8017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 09:00:02.858609 8240 raft_consensus.cc:2749] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260318 09:00:02.858683 7607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07f47abe644e4c6786bd63c1251a0ac2" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "e45b189483e74bc089b2963727d99490"
I20260318 09:00:02.860929 8014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 09:00:02.861593 8014 raft_consensus.cc:2468] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06af5a588953450dbb0862d99ddf80f0 in term 1.
I20260318 09:00:02.861080 8239 leader_election.cc:290] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [CANDIDATE]: Term 4 election: Requested vote from peers e45b189483e74bc089b2963727d99490 (127.6.2.126:40111), 1e52aab3abc24498a28a69e7c89452f9 (127.6.2.124:42683)
I20260318 09:00:02.863075 7826 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836; no voters:
I20260318 09:00:02.865031 7737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07f47abe644e4c6786bd63c1251a0ac2" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "1e52aab3abc24498a28a69e7c89452f9"
I20260318 09:00:02.865204 7940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 09:00:02.865594 7737 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 3 FOLLOWER]: Advancing to term 4
I20260318 09:00:02.858675 8017 raft_consensus.cc:2468] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 2.
I20260318 09:00:02.872813 7905 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836; no voters:
I20260318 09:00:02.875092 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 09:00:02.875254 8237 raft_consensus.cc:2804] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260318 09:00:02.875741 8237 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 09:00:02.875986 8237 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Advancing to term 3
I20260318 09:00:02.883373 8237 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 09:00:02.885258 7737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07f47abe644e4c6786bd63c1251a0ac2 in term 4.
I20260318 09:00:02.887331 8237 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 election: Requested vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 09:00:02.888095 8017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 09:00:02.888230 7645 leader_election.cc:304] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07f47abe644e4c6786bd63c1251a0ac2, 1e52aab3abc24498a28a69e7c89452f9; no voters: e45b189483e74bc089b2963727d99490
I20260318 09:00:02.889370 8239 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 4 FOLLOWER]: Leader election won for term 4
I20260318 09:00:02.892525 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0"
I20260318 09:00:02.894066 8236 raft_consensus.cc:2804] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260318 09:00:02.894522 8236 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d02c582e0d940cdbd7b66d2c951c9e1)
I20260318 09:00:02.894809 7906 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1; no voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836
I20260318 09:00:02.894907 8239 raft_consensus.cc:697] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 4 LEADER]: Becoming Leader. State: Replica: 07f47abe644e4c6786bd63c1251a0ac2, State: Running, Role: LEADER
I20260318 09:00:02.896032 8237 raft_consensus.cc:2749] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260318 09:00:02.894910 8236 raft_consensus.cc:3060] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 1 FOLLOWER]: Advancing to term 2
I20260318 09:00:02.895941 8239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 09:00:02.902235 7672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e52aab3abc24498a28a69e7c89452f9" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "07f47abe644e4c6786bd63c1251a0ac2" is_pre_election: true
I20260318 09:00:02.904220 7713 leader_election.cc:304] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e52aab3abc24498a28a69e7c89452f9; no voters: 07f47abe644e4c6786bd63c1251a0ac2, e45b189483e74bc089b2963727d99490
I20260318 09:00:02.904901 8236 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 09:00:02.905203 8238 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260318 09:00:02.907006 8236 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 election: Requested vote from peers 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 09:00:02.908022 8015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 09:00:02.909260 7737 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 4 FOLLOWER]: Rejecting Update request from peer e45b189483e74bc089b2963727d99490 for earlier term 3. Current term is 4. Ops: []
I20260318 09:00:02.911756 8244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07f47abe644e4c6786bd63c1251a0ac2. Latest consensus state: current_term: 4 leader_uuid: "07f47abe644e4c6786bd63c1251a0ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 09:00:02.912106 7861 raft_consensus.cc:1240] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Rejecting Update request from peer 3d02c582e0d940cdbd7b66d2c951c9e1 for earlier term 1. Current term is 2. Ops: []
I20260318 09:00:02.912742 8215 consensus_queue.cc:1059] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 }, Status: INVALID_TERM, Last received: 3.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260318 09:00:02.913064 8244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:02.913699 8231 consensus_queue.cc:1059] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 09:00:02.914907 8231 raft_consensus.cc:3055] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 LEADER]: Stepping down as leader of term 1
I20260318 09:00:02.915345 8231 raft_consensus.cc:740] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d02c582e0d940cdbd7b66d2c951c9e1, State: Running, Role: LEADER
I20260318 09:00:02.917517 7940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1"
I20260318 09:00:02.917431 8231 consensus_queue.cc:260] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 09:00:02.919663 8231 raft_consensus.cc:3060] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 1 FOLLOWER]: Advancing to term 2
I20260318 09:00:02.928848 8247 raft_consensus.cc:3055] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 3 LEADER]: Stepping down as leader of term 3
I20260318 09:00:02.929260 8247 raft_consensus.cc:740] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e45b189483e74bc089b2963727d99490, State: Running, Role: LEADER
I20260318 09:00:02.930023 8247 consensus_queue.cc:260] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 12, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } }
I20260318 09:00:02.931423 8247 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 3 FOLLOWER]: Advancing to term 4
I20260318 09:00:02.932682 8248 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 09:00:02.941437 7939 raft_consensus.cc:1240] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Rejecting Update request from peer 06af5a588953450dbb0862d99ddf80f0 for earlier term 2. Current term is 3. Ops: []
I20260318 09:00:02.943473 7940 raft_consensus.cc:2468] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06af5a588953450dbb0862d99ddf80f0 in term 2.
I20260318 09:00:02.946033 8236 consensus_queue.cc:1059] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260318 09:00:02.947365 8236 raft_consensus.cc:3055] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 2 LEADER]: Stepping down as leader of term 2
I20260318 09:00:02.947856 8236 raft_consensus.cc:740] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 06af5a588953450dbb0862d99ddf80f0, State: Running, Role: LEADER
I20260318 09:00:02.948704 8236 consensus_queue.cc:260] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
W20260318 09:00:02.931978 8246 consensus_queue.cc:1175] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "07f47abe644e4c6786bd63c1251a0ac2" responder_term: 4 status { last_received { term: 3 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e45b189483e74bc089b2963727d99490 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260318 09:00:02.949398 8248 catalog_manager.cc:679] Loaded metadata for table test-table [id=81693cd942f64504990e0adaa14b87f5]
I20260318 09:00:02.947405 7822 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1; no voters: 41e4ea383dc24ae5a6ef557c49dff836
I20260318 09:00:02.950172 8236 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Advancing to term 3
I20260318 09:00:02.951825 8228 raft_consensus.cc:2804] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260318 09:00:02.952283 8228 raft_consensus.cc:697] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 2 LEADER]: Becoming Leader. State: Replica: 06af5a588953450dbb0862d99ddf80f0, State: Running, Role: LEADER
I20260318 09:00:02.953049 8228 consensus_queue.cc:237] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 09:00:02.960065 8248 tablet_loader.cc:96] loaded metadata for tablet 0f3811d2d92f41c2b7301baa5ab3f1cb (table test-table [id=81693cd942f64504990e0adaa14b87f5])
I20260318 09:00:02.962473 8248 tablet_loader.cc:96] loaded metadata for tablet 5adac9d378be4afdbe6723195ec9bd11 (table test-table [id=81693cd942f64504990e0adaa14b87f5])
I20260318 09:00:02.964568 8248 tablet_loader.cc:96] loaded metadata for tablet a936692bfd8f4eaa9b58cb7cf64d05d4 (table test-table [id=81693cd942f64504990e0adaa14b87f5])
I20260318 09:00:02.965999 8248 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 09:00:02.970160 8248 catalog_manager.cc:1269] Loaded cluster ID: c582dd9307024b9483bfbf805eda4259
I20260318 09:00:02.970499 8248 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 09:00:02.974540 8248 catalog_manager.cc:1514] Loading token signing keys...
I20260318 09:00:02.977557 8248 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2: Loaded TSK: 0
I20260318 09:00:02.978334 8248 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 09:00:03.002841 7607 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 4 FOLLOWER]: Refusing update from remote peer 07f47abe644e4c6786bd63c1251a0ac2: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20260318 09:00:03.004060 7737 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 4 FOLLOWER]: Refusing update from remote peer 07f47abe644e4c6786bd63c1251a0ac2: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 4 index: 13. (index mismatch)
I20260318 09:00:03.004370 8244 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260318 09:00:03.005326 8239 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260318 09:00:03.010421 8238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07f47abe644e4c6786bd63c1251a0ac2. Latest consensus state: current_term: 4 leader_uuid: "07f47abe644e4c6786bd63c1251a0ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 09:00:03.011463 8238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 09:00:03.012161 8246 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07f47abe644e4c6786bd63c1251a0ac2. Latest consensus state: current_term: 4 leader_uuid: "07f47abe644e4c6786bd63c1251a0ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 09:00:03.012949 8246 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 09:00:03.016557 8244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "07f47abe644e4c6786bd63c1251a0ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 09:00:03.017521 8244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:03.017833 8238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "07f47abe644e4c6786bd63c1251a0ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 09:00:03.018636 8238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 09:00:03.018898 8244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "07f47abe644e4c6786bd63c1251a0ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 09:00:03.019659 8244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:03.021062 8246 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "07f47abe644e4c6786bd63c1251a0ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e45b189483e74bc089b2963727d99490" member_type: VOTER last_known_addr { host: "127.6.2.126" port: 40111 } } peers { permanent_uuid: "07f47abe644e4c6786bd63c1251a0ac2" member_type: VOTER last_known_addr { host: "127.6.2.125" port: 34659 } } peers { permanent_uuid: "1e52aab3abc24498a28a69e7c89452f9" member_type: VOTER last_known_addr { host: "127.6.2.124" port: 42683 } } }
I20260318 09:00:03.021751 8246 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [sys.catalog]: This master's current role is: FOLLOWER
I20260318 09:00:03.057380 7940 raft_consensus.cc:1275] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Refusing update from remote peer 06af5a588953450dbb0862d99ddf80f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260318 09:00:03.058735 8228 consensus_queue.cc:1048] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 09:00:03.061638 7966 heartbeater.cc:499] Master 127.6.2.125:34659 was elected leader, sending a full tablet report...
I20260318 09:00:03.075894 8015 raft_consensus.cc:3060] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 1 FOLLOWER]: Advancing to term 2
I20260318 09:00:03.078444 7660 catalog_manager.cc:5671] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 reported cstate change: term changed from 1 to 2, leader changed from 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66) to 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65). New cstate: current_term: 2 leader_uuid: "06af5a588953450dbb0862d99ddf80f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } }
I20260318 09:00:03.086593 7888 heartbeater.cc:499] Master 127.6.2.125:34659 was elected leader, sending a full tablet report...
I20260318 09:00:03.087672 8015 raft_consensus.cc:1275] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Refusing update from remote peer 06af5a588953450dbb0862d99ddf80f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260318 09:00:03.090520 8249 consensus_queue.cc:1048] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260318 09:00:03.098583 8044 heartbeater.cc:499] Master 127.6.2.125:34659 was elected leader, sending a full tablet report...
I20260318 09:00:03.127259 8228 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 09:00:03.127632 8228 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 09:00:03.128978 8228 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 09:00:03.129770 7940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 09:00:03.129946 8015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 09:00:03.130334 7940 raft_consensus.cc:2468] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06af5a588953450dbb0862d99ddf80f0 in term 3.
I20260318 09:00:03.131508 7822 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1; no voters: 41e4ea383dc24ae5a6ef557c49dff836
I20260318 09:00:03.132226 8228 raft_consensus.cc:2804] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260318 09:00:03.132578 8228 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 09:00:03.132902 8228 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Advancing to term 4
I20260318 09:00:03.134635 7778 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260318 09:00:03.138240 8228 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 09:00:03.139742 8228 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 09:00:03.140681 7940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1"
I20260318 09:00:03.140903 8015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "06af5a588953450dbb0862d99ddf80f0" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 09:00:03.141278 7940 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Advancing to term 4
I20260318 09:00:03.146449 7940 raft_consensus.cc:2468] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06af5a588953450dbb0862d99ddf80f0 in term 4.
I20260318 09:00:03.147555 7822 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1; no voters: 41e4ea383dc24ae5a6ef557c49dff836
I20260318 09:00:03.148196 8228 raft_consensus.cc:2804] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 4 FOLLOWER]: Leader election won for term 4
I20260318 09:00:03.148736 8242 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 09:00:03.148651 8228 raft_consensus.cc:697] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 4 LEADER]: Becoming Leader. State: Replica: 06af5a588953450dbb0862d99ddf80f0, State: Running, Role: LEADER
I20260318 09:00:03.149250 8242 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 09:00:03.150030 8228 consensus_queue.cc:237] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 09:00:03.151420 8242 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
I20260318 09:00:03.151958 7861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "06af5a588953450dbb0862d99ddf80f0" is_pre_election: true
I20260318 09:00:03.152866 7940 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "41e4ea383dc24ae5a6ef557c49dff836" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" is_pre_election: true
I20260318 09:00:03.153589 7940 raft_consensus.cc:2368] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41e4ea383dc24ae5a6ef557c49dff836 for earlier term 3. Current term is 4.
I20260318 09:00:03.154516 7981 leader_election.cc:400] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207) with higher term. Message: Invalid argument: T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41e4ea383dc24ae5a6ef557c49dff836 for earlier term 3. Current term is 4.
I20260318 09:00:03.154865 7981 leader_election.cc:403] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260318 09:00:03.155383 8242 raft_consensus.cc:3060] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Advancing to term 4
I20260318 09:00:03.158249 7660 catalog_manager.cc:5671] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "06af5a588953450dbb0862d99ddf80f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } health_report { overall_health: UNKNOWN } } }
I20260318 09:00:03.162935 8242 raft_consensus.cc:2749] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207) with higher term. Message: Invalid argument: T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 41e4ea383dc24ae5a6ef557c49dff836 for earlier term 3. Current term is 4.
I20260318 09:00:03.238646 8015 raft_consensus.cc:1275] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 4 FOLLOWER]: Refusing update from remote peer 06af5a588953450dbb0862d99ddf80f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260318 09:00:03.239954 8236 consensus_queue.cc:1048] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260318 09:00:03.256850 7940 raft_consensus.cc:1275] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 4 FOLLOWER]: Refusing update from remote peer 06af5a588953450dbb0862d99ddf80f0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260318 09:00:03.259914 8228 consensus_queue.cc:1048] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260318 09:00:03.849112 8039 debug-util.cc:398] Leaking SignalData structure 0x7b080020eac0 after lost signal to thread 6156
W20260318 09:00:03.849975 8039 debug-util.cc:398] Leaking SignalData structure 0x7b080020f220 after lost signal to thread 7631
W20260318 09:00:03.850731 8039 debug-util.cc:398] Leaking SignalData structure 0x7b080020f8e0 after lost signal to thread 7696
W20260318 09:00:03.851644 8039 debug-util.cc:398] Leaking SignalData structure 0x7b0800138000 after lost signal to thread 7761
W20260318 09:00:03.852735 8039 debug-util.cc:398] Leaking SignalData structure 0x7b08000f4d40 after lost signal to thread 7886
W20260318 09:00:03.853739 8039 debug-util.cc:398] Leaking SignalData structure 0x7b08001353e0 after lost signal to thread 7964
W20260318 09:00:03.854823 8039 debug-util.cc:398] Leaking SignalData structure 0x7b080009f3a0 after lost signal to thread 8042
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:784: Failure
Expected equality of these values:
live_row_count
Which is: 929
table_info->GetMetrics()->live_row_count->value()
Which is: 0
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 784, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:823: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260318 09:00:05.213452 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
W20260318 09:00:05.257658 7906 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260318 09:00:05.261237 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260318 09:00:05.261654 7906 consensus_peers.cc:597] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 -> Peer 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351): Couldn't send request to peer 06af5a588953450dbb0862d99ddf80f0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260318 09:00:05.264472 6153 tablet_replica.cc:333] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0: stopping tablet replica
I20260318 09:00:05.265518 6153 raft_consensus.cc:2243] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 4 LEADER]: Raft consensus shutting down.
I20260318 09:00:05.266945 6153 raft_consensus.cc:2272] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 06af5a588953450dbb0862d99ddf80f0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:05.270807 6153 tablet_replica.cc:333] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0: stopping tablet replica
I20260318 09:00:05.271595 6153 raft_consensus.cc:2243] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:05.272153 6153 raft_consensus.cc:2272] T 5adac9d378be4afdbe6723195ec9bd11 P 06af5a588953450dbb0862d99ddf80f0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:05.275871 6153 tablet_replica.cc:333] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0: stopping tablet replica
I20260318 09:00:05.276643 6153 raft_consensus.cc:2243] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 2 LEADER]: Raft consensus shutting down.
I20260318 09:00:05.277704 6153 raft_consensus.cc:2272] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 06af5a588953450dbb0862d99ddf80f0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:05.342818 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 09:00:05.356264 8237 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 09:00:05.356889 8237 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 09:00:05.364773 8017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a936692bfd8f4eaa9b58cb7cf64d05d4" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 09:00:05.366907 8237 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185)
I20260318 09:00:05.380086 6153 tablet_server.cc:178] TabletServer@127.6.2.66:0 shutting down...
W20260318 09:00:05.383328 7906 leader_election.cc:336] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351): Network error: Client connection negotiation failed: client connection to 127.6.2.65:35351: connect: Connection refused (error 111)
I20260318 09:00:05.383971 7906 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1; no voters: 06af5a588953450dbb0862d99ddf80f0, 41e4ea383dc24ae5a6ef557c49dff836
I20260318 09:00:05.384984 8273 raft_consensus.cc:2749] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260318 09:00:05.394882 8275 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 09:00:05.398159 8275 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 09:00:05.400627 8275 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207), 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351)
I20260318 09:00:05.400749 8276 raft_consensus.cc:493] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 09:00:05.401492 8276 raft_consensus.cc:515] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } }
I20260318 09:00:05.407076 8276 leader_election.cc:290] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351), 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207)
W20260318 09:00:05.411175 7981 leader_election.cc:336] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260318 09:00:05.415998 7981 leader_election.cc:336] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 3d02c582e0d940cdbd7b66d2c951c9e1 (127.6.2.66:45207): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260318 09:00:05.419824 7984 leader_election.cc:336] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351): Network error: Client connection negotiation failed: client connection to 127.6.2.65:35351: connect: Connection refused (error 111)
I20260318 09:00:05.420289 7984 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1
I20260318 09:00:05.422334 8276 raft_consensus.cc:2749] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260318 09:00:05.422446 7984 leader_election.cc:336] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351): Network error: Client connection negotiation failed: client connection to 127.6.2.65:35351: connect: Connection refused (error 111)
I20260318 09:00:05.423059 7984 leader_election.cc:304] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0, 3d02c582e0d940cdbd7b66d2c951c9e1
I20260318 09:00:05.424388 8276 raft_consensus.cc:2749] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260318 09:00:05.424865 8273 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 09:00:05.425384 8273 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 09:00:05.428170 8273 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185), 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351)
I20260318 09:00:05.429005 8017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836" is_pre_election: true
I20260318 09:00:05.429523 8017 raft_consensus.cc:2468] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 2.
I20260318 09:00:05.430552 7905 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836; no voters:
I20260318 09:00:05.431625 8273 raft_consensus.cc:2804] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260318 09:00:05.432113 8273 raft_consensus.cc:493] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 06af5a588953450dbb0862d99ddf80f0)
I20260318 09:00:05.432245 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:05.432657 8273 raft_consensus.cc:3060] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 2 FOLLOWER]: Advancing to term 3
W20260318 09:00:05.433880 7906 leader_election.cc:336] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351): Network error: Client connection negotiation failed: client connection to 127.6.2.65:35351: connect: Connection refused (error 111)
I20260318 09:00:05.434942 6153 tablet_replica.cc:333] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1: stopping tablet replica
I20260318 09:00:05.439102 8273 raft_consensus.cc:515] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41e4ea383dc24ae5a6ef557c49dff836" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45185 } } peers { permanent_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 45207 } } peers { permanent_uuid: "06af5a588953450dbb0862d99ddf80f0" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35351 } }
I20260318 09:00:05.440068 6153 raft_consensus.cc:2243] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:05.441991 8017 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f3811d2d92f41c2b7301baa5ab3f1cb" candidate_uuid: "3d02c582e0d940cdbd7b66d2c951c9e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "41e4ea383dc24ae5a6ef557c49dff836"
I20260318 09:00:05.442560 8017 raft_consensus.cc:3060] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 2 FOLLOWER]: Advancing to term 3
I20260318 09:00:05.443653 6153 raft_consensus.cc:2272] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:05.445788 8273 leader_election.cc:290] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 election: Requested vote from peers 41e4ea383dc24ae5a6ef557c49dff836 (127.6.2.67:45185), 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351)
W20260318 09:00:05.448439 7906 leader_election.cc:336] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 06af5a588953450dbb0862d99ddf80f0 (127.6.2.65:35351): Network error: Client connection negotiation failed: client connection to 127.6.2.65:35351: connect: Connection refused (error 111)
I20260318 09:00:05.449573 8017 raft_consensus.cc:2468] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d02c582e0d940cdbd7b66d2c951c9e1 in term 3.
I20260318 09:00:05.451805 7905 leader_election.cc:304] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d02c582e0d940cdbd7b66d2c951c9e1, 41e4ea383dc24ae5a6ef557c49dff836; no voters: 06af5a588953450dbb0862d99ddf80f0
W20260318 09:00:05.452548 7905 raft_consensus.cc:2686] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 3d02c582e0d940cdbd7b66d2c951c9e1: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260318 09:00:05.455924 6153 tablet_replica.cc:333] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1: stopping tablet replica
I20260318 09:00:05.456759 6153 raft_consensus.cc:2243] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:05.457382 6153 raft_consensus.cc:2272] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:05.461035 6153 tablet_replica.cc:333] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1: stopping tablet replica
I20260318 09:00:05.461696 6153 raft_consensus.cc:2243] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 LEADER]: Raft consensus shutting down.
I20260318 09:00:05.462716 6153 raft_consensus.cc:2272] T 5adac9d378be4afdbe6723195ec9bd11 P 3d02c582e0d940cdbd7b66d2c951c9e1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:05.491792 6153 tablet_server.cc:195] TabletServer@127.6.2.66:0 shutdown complete.
I20260318 09:00:05.506574 6153 tablet_server.cc:178] TabletServer@127.6.2.67:0 shutting down...
I20260318 09:00:05.545918 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:05.546684 6153 tablet_replica.cc:333] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836: stopping tablet replica
I20260318 09:00:05.547598 6153 raft_consensus.cc:2243] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:05.548195 6153 raft_consensus.cc:2272] T 0f3811d2d92f41c2b7301baa5ab3f1cb P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:05.550789 6153 tablet_replica.cc:333] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836: stopping tablet replica
I20260318 09:00:05.551378 6153 raft_consensus.cc:2243] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:05.551896 6153 raft_consensus.cc:2272] T 5adac9d378be4afdbe6723195ec9bd11 P 41e4ea383dc24ae5a6ef557c49dff836 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:05.554306 6153 tablet_replica.cc:333] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836: stopping tablet replica
I20260318 09:00:05.554802 6153 raft_consensus.cc:2243] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:05.555337 6153 raft_consensus.cc:2272] T a936692bfd8f4eaa9b58cb7cf64d05d4 P 41e4ea383dc24ae5a6ef557c49dff836 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:06.578408 6153 tablet_server.cc:195] TabletServer@127.6.2.67:0 shutdown complete.
I20260318 09:00:06.598623 6153 master.cc:561] Master@127.6.2.126:40111 shutting down...
I20260318 09:00:06.625924 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:06.626590 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:06.626924 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P e45b189483e74bc089b2963727d99490: stopping tablet replica
I20260318 09:00:06.676256 6153 master.cc:583] Master@127.6.2.126:40111 shutdown complete.
W20260318 09:00:06.676468 7648 consensus_peers.cc:597] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 -> Peer e45b189483e74bc089b2963727d99490 (127.6.2.126:40111): Couldn't send request to peer e45b189483e74bc089b2963727d99490. Status: Network error: Client connection negotiation failed: client connection to 127.6.2.126:40111: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260318 09:00:06.697100 6153 master.cc:561] Master@127.6.2.125:34659 shutting down...
I20260318 09:00:06.721463 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 4 LEADER]: Raft consensus shutting down.
I20260318 09:00:06.722445 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:06.722827 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 07f47abe644e4c6786bd63c1251a0ac2: stopping tablet replica
I20260318 09:00:06.759742 6153 master.cc:583] Master@127.6.2.125:34659 shutdown complete.
I20260318 09:00:06.777777 6153 master.cc:561] Master@127.6.2.124:42683 shutting down...
I20260318 09:00:06.798300 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:06.798906 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:06.799305 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 1e52aab3abc24498a28a69e7c89452f9: stopping tablet replica
I20260318 09:00:06.818806 6153 master.cc:583] Master@127.6.2.124:42683 shutdown complete.
I20260318 09:00:06.848423 6153 test_util.cc:183] -----------------------------------------------
I20260318 09:00:06.848690 6153 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1773824355427976-6153-0
[ FAILED ] TsTabletManagerITest.TestTableStats (36415 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20260318 09:00:06.855976 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:38655
I20260318 09:00:06.857129 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:06.863744 8288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:06.862711 8287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:06.865226 8290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:06.867619 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:06.868827 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:06.869017 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:06.869131 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824406869120 us; error 0 us; skew 500 ppm
I20260318 09:00:06.869613 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:06.872045 6153 webserver.cc:492] Webserver started at http://127.6.2.126:40517/ using document root <none> and password file <none>
I20260318 09:00:06.872502 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:06.872659 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:06.872867 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:06.874076 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "8eb6b42bb0814254a45b17c07d9d2f1e"
format_stamp: "Formatted at 2026-03-18 09:00:06 on dist-test-slave-mpvv"
I20260318 09:00:06.878556 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260318 09:00:06.881953 8295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:06.882742 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260318 09:00:06.883036 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "8eb6b42bb0814254a45b17c07d9d2f1e"
format_stamp: "Formatted at 2026-03-18 09:00:06 on dist-test-slave-mpvv"
I20260318 09:00:06.883395 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:06.896512 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:06.897768 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:06.938073 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:38655
I20260318 09:00:06.938194 8346 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:38655 every 8 connection(s)
I20260318 09:00:06.942193 8347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:06.952522 8347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e: Bootstrap starting.
I20260318 09:00:06.957264 8347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:06.961419 8347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e: No bootstrap required, opened a new log
I20260318 09:00:06.963663 8347 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb6b42bb0814254a45b17c07d9d2f1e" member_type: VOTER }
I20260318 09:00:06.964075 8347 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:06.964255 8347 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb6b42bb0814254a45b17c07d9d2f1e, State: Initialized, Role: FOLLOWER
I20260318 09:00:06.964737 8347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb6b42bb0814254a45b17c07d9d2f1e" member_type: VOTER }
I20260318 09:00:06.965168 8347 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:06.965358 8347 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:06.965597 8347 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:06.970265 8347 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb6b42bb0814254a45b17c07d9d2f1e" member_type: VOTER }
I20260318 09:00:06.970772 8347 leader_election.cc:304] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eb6b42bb0814254a45b17c07d9d2f1e; no voters:
I20260318 09:00:06.972009 8347 leader_election.cc:290] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:06.972486 8350 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:06.973986 8350 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 1 LEADER]: Becoming Leader. State: Replica: 8eb6b42bb0814254a45b17c07d9d2f1e, State: Running, Role: LEADER
I20260318 09:00:06.974709 8350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb6b42bb0814254a45b17c07d9d2f1e" member_type: VOTER }
I20260318 09:00:06.975020 8347 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [sys.catalog]: configured and running, proceeding with master startup.
I20260318 09:00:06.980443 8352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb6b42bb0814254a45b17c07d9d2f1e. Latest consensus state: current_term: 1 leader_uuid: "8eb6b42bb0814254a45b17c07d9d2f1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb6b42bb0814254a45b17c07d9d2f1e" member_type: VOTER } }
I20260318 09:00:06.981096 8352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:06.982340 8351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eb6b42bb0814254a45b17c07d9d2f1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb6b42bb0814254a45b17c07d9d2f1e" member_type: VOTER } }
I20260318 09:00:06.983029 8351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:06.991356 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20260318 09:00:06.994052 8366 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260318 09:00:06.994417 8366 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260318 09:00:06.994848 8358 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 09:00:06.999488 8358 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 09:00:07.008585 8358 catalog_manager.cc:1357] Generated new cluster ID: 1d0e09be7e5e49f398f7f3f33209cd4c
I20260318 09:00:07.008946 8358 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 09:00:07.028445 8358 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 09:00:07.029765 8358 catalog_manager.cc:1514] Loading token signing keys...
I20260318 09:00:07.045818 8358 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e: Generated new TSK 0
I20260318 09:00:07.046414 8358 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 09:00:07.057996 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:07.064265 8368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:07.065289 8369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:07.067624 6153 server_base.cc:1060] running on GCE node
W20260318 09:00:07.068104 8371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:07.069024 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:07.069273 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:07.069464 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824407069445 us; error 0 us; skew 500 ppm
I20260318 09:00:07.070039 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:07.072430 6153 webserver.cc:492] Webserver started at http://127.6.2.65:42923/ using document root <none> and password file <none>
I20260318 09:00:07.072945 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:07.073161 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:07.073446 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:07.074528 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "8324d6c954fa44908364578b4cd81a3f"
format_stamp: "Formatted at 2026-03-18 09:00:07 on dist-test-slave-mpvv"
I20260318 09:00:07.079236 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260318 09:00:07.082643 8376 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:07.083588 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260318 09:00:07.083844 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "8324d6c954fa44908364578b4cd81a3f"
format_stamp: "Formatted at 2026-03-18 09:00:07 on dist-test-slave-mpvv"
I20260318 09:00:07.084092 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:07.099064 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:07.100632 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:07.102061 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 09:00:07.104396 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:07.104578 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:07.104768 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:07.104890 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:07.145905 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:33873
I20260318 09:00:07.146005 8438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:33873 every 8 connection(s)
I20260318 09:00:07.150872 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:07.158598 8443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:07.160784 8444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:07.162830 8446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:07.163221 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:07.164181 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:07.164378 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:07.164588 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824407164573 us; error 0 us; skew 500 ppm
I20260318 09:00:07.165172 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:07.168044 6153 webserver.cc:492] Webserver started at http://127.6.2.66:36409/ using document root <none> and password file <none>
I20260318 09:00:07.168744 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:07.168993 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:07.169327 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:07.169291 8439 heartbeater.cc:344] Connected to a master server at 127.6.2.126:38655
I20260318 09:00:07.169996 8439 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:07.170742 8439 heartbeater.cc:507] Master 127.6.2.126:38655 requested a full tablet report, sending...
I20260318 09:00:07.170753 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-1-root/instance:
uuid: "a0c912cb1f164f3c99ba56f923ac350f"
format_stamp: "Formatted at 2026-03-18 09:00:07 on dist-test-slave-mpvv"
I20260318 09:00:07.173297 8312 ts_manager.cc:194] Registered new tserver with Master: 8324d6c954fa44908364578b4cd81a3f (127.6.2.65:33873)
I20260318 09:00:07.175770 8312 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48008
I20260318 09:00:07.177083 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260318 09:00:07.180752 8451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:07.181571 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260318 09:00:07.181902 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-1-root
uuid: "a0c912cb1f164f3c99ba56f923ac350f"
format_stamp: "Formatted at 2026-03-18 09:00:07 on dist-test-slave-mpvv"
I20260318 09:00:07.182157 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:07.307516 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:07.308830 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:07.310336 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 09:00:07.312634 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:07.312875 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:07.313143 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:07.313334 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:07.354193 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.66:35601
I20260318 09:00:07.354303 8513 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.66:35601 every 8 connection(s)
I20260318 09:00:07.358955 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:07.366799 8517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:07.368317 8518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:07.369750 8520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:07.370321 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:07.371425 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:07.371626 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:07.371867 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824407371848 us; error 0 us; skew 500 ppm
I20260318 09:00:07.372391 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:07.374131 8514 heartbeater.cc:344] Connected to a master server at 127.6.2.126:38655
I20260318 09:00:07.374490 8514 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:07.375345 8514 heartbeater.cc:507] Master 127.6.2.126:38655 requested a full tablet report, sending...
I20260318 09:00:07.375357 6153 webserver.cc:492] Webserver started at http://127.6.2.67:36253/ using document root <none> and password file <none>
I20260318 09:00:07.376019 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:07.376264 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:07.376608 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:07.377487 8312 ts_manager.cc:194] Registered new tserver with Master: a0c912cb1f164f3c99ba56f923ac350f (127.6.2.66:35601)
I20260318 09:00:07.377873 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-2-root/instance:
uuid: "cf17c8d0b3e34fd28d632d53f9d7153b"
format_stamp: "Formatted at 2026-03-18 09:00:07 on dist-test-slave-mpvv"
I20260318 09:00:07.378917 8312 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48022
I20260318 09:00:07.382417 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260318 09:00:07.386101 8525 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:07.386861 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260318 09:00:07.387228 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-2-root
uuid: "cf17c8d0b3e34fd28d632d53f9d7153b"
format_stamp: "Formatted at 2026-03-18 09:00:07 on dist-test-slave-mpvv"
I20260318 09:00:07.387528 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:07.419301 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:07.420545 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:07.422025 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 09:00:07.424315 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:07.424557 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:07.424820 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:07.425009 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:07.466667 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.67:35139
I20260318 09:00:07.466774 8587 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.67:35139 every 8 connection(s)
I20260318 09:00:07.480448 8588 heartbeater.cc:344] Connected to a master server at 127.6.2.126:38655
I20260318 09:00:07.480832 8588 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:07.481681 8588 heartbeater.cc:507] Master 127.6.2.126:38655 requested a full tablet report, sending...
I20260318 09:00:07.483542 8312 ts_manager.cc:194] Registered new tserver with Master: cf17c8d0b3e34fd28d632d53f9d7153b (127.6.2.67:35139)
I20260318 09:00:07.483716 6153 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013674123s
I20260318 09:00:07.485419 8312 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48026
I20260318 09:00:07.517362 8312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260318 09:00:07.518855 8312 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260318 09:00:07.561682 8553 tablet_service.cc:1511] Processing CreateTablet for tablet c6dcb4f32a544b738afa5da77805c16a (DEFAULT_TABLE table=test-table [id=a72231a9928e4591a75143a413d0b034]), partition=
I20260318 09:00:07.562980 8553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6dcb4f32a544b738afa5da77805c16a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:07.564867 8479 tablet_service.cc:1511] Processing CreateTablet for tablet c6dcb4f32a544b738afa5da77805c16a (DEFAULT_TABLE table=test-table [id=a72231a9928e4591a75143a413d0b034]), partition=
I20260318 09:00:07.566054 8479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6dcb4f32a544b738afa5da77805c16a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:07.573836 8404 tablet_service.cc:1511] Processing CreateTablet for tablet c6dcb4f32a544b738afa5da77805c16a (DEFAULT_TABLE table=test-table [id=a72231a9928e4591a75143a413d0b034]), partition=
I20260318 09:00:07.575013 8404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6dcb4f32a544b738afa5da77805c16a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:07.580793 8604 tablet_bootstrap.cc:492] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: Bootstrap starting.
I20260318 09:00:07.584628 8605 tablet_bootstrap.cc:492] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: Bootstrap starting.
I20260318 09:00:07.588186 8604 tablet_bootstrap.cc:654] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:07.591005 8605 tablet_bootstrap.cc:654] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:07.593590 8604 tablet_bootstrap.cc:492] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: No bootstrap required, opened a new log
I20260318 09:00:07.594038 8604 ts_tablet_manager.cc:1403] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260318 09:00:07.595032 8607 tablet_bootstrap.cc:492] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Bootstrap starting.
I20260318 09:00:07.596455 8605 tablet_bootstrap.cc:492] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: No bootstrap required, opened a new log
I20260318 09:00:07.596930 8605 ts_tablet_manager.cc:1403] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260318 09:00:07.596941 8604 raft_consensus.cc:359] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
I20260318 09:00:07.597852 8604 raft_consensus.cc:385] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:07.598213 8604 raft_consensus.cc:740] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf17c8d0b3e34fd28d632d53f9d7153b, State: Initialized, Role: FOLLOWER
I20260318 09:00:07.598904 8604 consensus_queue.cc:260] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
I20260318 09:00:07.601500 8588 heartbeater.cc:499] Master 127.6.2.126:38655 was elected leader, sending a full tablet report...
I20260318 09:00:07.601959 8604 ts_tablet_manager.cc:1434] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260318 09:00:07.601673 8605 raft_consensus.cc:359] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
W20260318 09:00:07.602252 8604 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260318 09:00:07.602427 8605 raft_consensus.cc:385] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:07.602814 8605 raft_consensus.cc:740] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c912cb1f164f3c99ba56f923ac350f, State: Initialized, Role: FOLLOWER
I20260318 09:00:07.603235 8607 tablet_bootstrap.cc:654] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:07.603605 8605 consensus_queue.cc:260] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
I20260318 09:00:07.605887 8514 heartbeater.cc:499] Master 127.6.2.126:38655 was elected leader, sending a full tablet report...
I20260318 09:00:07.606546 8605 ts_tablet_manager.cc:1434] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260318 09:00:07.606832 8605 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
W20260318 09:00:07.608561 8515 tablet.cc:2379] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260318 09:00:07.612514 8607 tablet_bootstrap.cc:492] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: No bootstrap required, opened a new log
I20260318 09:00:07.613049 8607 ts_tablet_manager.cc:1403] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20260318 09:00:07.615272 8607 raft_consensus.cc:359] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
I20260318 09:00:07.615847 8607 raft_consensus.cc:385] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:07.616094 8607 raft_consensus.cc:740] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8324d6c954fa44908364578b4cd81a3f, State: Initialized, Role: FOLLOWER
I20260318 09:00:07.616752 8607 consensus_queue.cc:260] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
I20260318 09:00:07.618672 8439 heartbeater.cc:499] Master 127.6.2.126:38655 was elected leader, sending a full tablet report...
I20260318 09:00:07.619020 8607 ts_tablet_manager.cc:1434] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260318 09:00:07.619349 8607 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260318 09:00:07.811786 8609 raft_consensus.cc:493] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 09:00:07.812310 8609 raft_consensus.cc:515] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
I20260318 09:00:07.814424 8609 leader_election.cc:290] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0c912cb1f164f3c99ba56f923ac350f (127.6.2.66:35601), 8324d6c954fa44908364578b4cd81a3f (127.6.2.65:33873)
I20260318 09:00:07.823246 8489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6dcb4f32a544b738afa5da77805c16a" candidate_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0c912cb1f164f3c99ba56f923ac350f" is_pre_election: true
I20260318 09:00:07.823957 8489 raft_consensus.cc:2468] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf17c8d0b3e34fd28d632d53f9d7153b in term 0.
I20260318 09:00:07.825035 8414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6dcb4f32a544b738afa5da77805c16a" candidate_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8324d6c954fa44908364578b4cd81a3f" is_pre_election: true
I20260318 09:00:07.825057 8527 leader_election.cc:304] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c912cb1f164f3c99ba56f923ac350f, cf17c8d0b3e34fd28d632d53f9d7153b; no voters:
I20260318 09:00:07.825773 8414 raft_consensus.cc:2468] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf17c8d0b3e34fd28d632d53f9d7153b in term 0.
I20260318 09:00:07.825977 8609 raft_consensus.cc:2804] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260318 09:00:07.826385 8609 raft_consensus.cc:493] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 09:00:07.826772 8609 raft_consensus.cc:3060] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:07.831931 8609 raft_consensus.cc:515] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
I20260318 09:00:07.833482 8609 leader_election.cc:290] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [CANDIDATE]: Term 1 election: Requested vote from peers a0c912cb1f164f3c99ba56f923ac350f (127.6.2.66:35601), 8324d6c954fa44908364578b4cd81a3f (127.6.2.65:33873)
I20260318 09:00:07.834303 8489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6dcb4f32a544b738afa5da77805c16a" candidate_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0c912cb1f164f3c99ba56f923ac350f"
I20260318 09:00:07.834517 8414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6dcb4f32a544b738afa5da77805c16a" candidate_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8324d6c954fa44908364578b4cd81a3f"
I20260318 09:00:07.834860 8489 raft_consensus.cc:3060] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:07.835078 8414 raft_consensus.cc:3060] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:07.839970 8489 raft_consensus.cc:2468] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf17c8d0b3e34fd28d632d53f9d7153b in term 1.
I20260318 09:00:07.839970 8414 raft_consensus.cc:2468] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf17c8d0b3e34fd28d632d53f9d7153b in term 1.
I20260318 09:00:07.841114 8527 leader_election.cc:304] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c912cb1f164f3c99ba56f923ac350f, cf17c8d0b3e34fd28d632d53f9d7153b; no voters:
I20260318 09:00:07.841840 8609 raft_consensus.cc:2804] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:07.842687 8609 raft_consensus.cc:697] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 1 LEADER]: Becoming Leader. State: Replica: cf17c8d0b3e34fd28d632d53f9d7153b, State: Running, Role: LEADER
I20260318 09:00:07.843513 8609 consensus_queue.cc:237] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
I20260318 09:00:07.851737 8311 catalog_manager.cc:5671] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b reported cstate change: term changed from 0 to 1, leader changed from <none> to cf17c8d0b3e34fd28d632d53f9d7153b (127.6.2.67). New cstate: current_term: 1 leader_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } health_report { overall_health: UNKNOWN } } }
I20260318 09:00:07.913301 8404 tablet_service.cc:1558] Processing DeleteTablet for tablet c6dcb4f32a544b738afa5da77805c16a with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:43202
I20260318 09:00:07.914819 8617 tablet_replica.cc:333] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: stopping tablet replica
I20260318 09:00:07.915513 8617 raft_consensus.cc:2243] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:07.915997 8617 raft_consensus.cc:2272] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:07.918141 8617 ts_tablet_manager.cc:1916] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260318 09:00:07.929248 8617 ts_tablet_manager.cc:1929] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20260318 09:00:07.929661 8617 log.cc:1199] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Deleting WAL directory at /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-0-root/wals/c6dcb4f32a544b738afa5da77805c16a
I20260318 09:00:07.933699 8618 ts_tablet_manager.cc:933] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Initiating tablet copy from peer cf17c8d0b3e34fd28d632d53f9d7153b (127.6.2.67:35139)
I20260318 09:00:07.935563 8618 tablet_copy_client.cc:323] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.2.67:35139
I20260318 09:00:07.945360 8573 tablet_copy_service.cc:140] P cf17c8d0b3e34fd28d632d53f9d7153b: Received BeginTabletCopySession request for tablet c6dcb4f32a544b738afa5da77805c16a from peer 8324d6c954fa44908364578b4cd81a3f ({username='slave'} at 127.0.0.1:56344)
I20260318 09:00:07.945892 8573 tablet_copy_service.cc:161] P cf17c8d0b3e34fd28d632d53f9d7153b: Beginning new tablet copy session on tablet c6dcb4f32a544b738afa5da77805c16a from peer 8324d6c954fa44908364578b4cd81a3f at {username='slave'} at 127.0.0.1:56344: session id = 8324d6c954fa44908364578b4cd81a3f-c6dcb4f32a544b738afa5da77805c16a
I20260318 09:00:07.951501 8573 tablet_copy_source_session.cc:215] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: Tablet Copy: opened 0 blocks and 1 log segments
I20260318 09:00:07.954087 8618 ts_tablet_manager.cc:1916] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260318 09:00:07.964049 8618 ts_tablet_manager.cc:1929] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20260318 09:00:07.964607 8618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6dcb4f32a544b738afa5da77805c16a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:07.969597 8618 tablet_copy_client.cc:806] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: tablet copy: Starting download of 0 data blocks...
I20260318 09:00:07.970151 8618 tablet_copy_client.cc:670] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: tablet copy: Starting download of 1 WAL segments...
I20260318 09:00:07.971886 8311 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48042:
table { table_id: "a72231a9928e4591a75143a413d0b034" table_name: "test-table" }
I20260318 09:00:07.972607 8311 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48042:
table { table_id: "a72231a9928e4591a75143a413d0b034" table_name: "test-table" }
I20260318 09:00:07.974850 8618 tablet_copy_client.cc:538] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260318 09:00:07.985188 8618 tablet_bootstrap.cc:492] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Bootstrap starting.
I20260318 09:00:07.985940 8311 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e: Sending DeleteTablet for 3 replicas of tablet c6dcb4f32a544b738afa5da77805c16a
I20260318 09:00:07.988240 8479 tablet_service.cc:1558] Processing DeleteTablet for tablet c6dcb4f32a544b738afa5da77805c16a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-18 09:00:07 UTC) from {username='slave'} at 127.0.0.1:59012
I20260318 09:00:07.988133 8553 tablet_service.cc:1558] Processing DeleteTablet for tablet c6dcb4f32a544b738afa5da77805c16a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-18 09:00:07 UTC) from {username='slave'} at 127.0.0.1:56328
I20260318 09:00:07.990340 8404 tablet_service.cc:1558] Processing DeleteTablet for tablet c6dcb4f32a544b738afa5da77805c16a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-18 09:00:07 UTC) from {username='slave'} at 127.0.0.1:43192
W20260318 09:00:07.992411 8297 catalog_manager.cc:4982] TS 8324d6c954fa44908364578b4cd81a3f (127.6.2.65:33873): delete failed for tablet c6dcb4f32a544b738afa5da77805c16a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6dcb4f32a544b738afa5da77805c16a already in progress: copying tablet
I20260318 09:00:07.996306 8622 tablet_replica.cc:333] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: stopping tablet replica
I20260318 09:00:07.997701 8622 raft_consensus.cc:2243] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:07.998584 8622 pending_rounds.cc:70] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: Trying to abort 1 pending ops.
I20260318 09:00:07.998952 8622 pending_rounds.cc:77] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265584774533484544 op_type: NO_OP noop_request { }
I20260318 09:00:07.999437 8622 raft_consensus.cc:2889] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260318 09:00:07.999843 8622 raft_consensus.cc:2272] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:08.001004 8621 tablet_replica.cc:333] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: stopping tablet replica
I20260318 09:00:08.001703 8621 raft_consensus.cc:2243] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:08.002193 8621 raft_consensus.cc:2272] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:08.003046 8622 ts_tablet_manager.cc:1916] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260318 09:00:08.003429 8618 tablet_bootstrap.cc:492] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260318 09:00:08.004408 8618 tablet_bootstrap.cc:492] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Bootstrap complete.
I20260318 09:00:08.004448 8621 ts_tablet_manager.cc:1916] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260318 09:00:08.005205 8618 ts_tablet_manager.cc:1403] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20260318 09:00:08.008241 8618 raft_consensus.cc:359] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
I20260318 09:00:08.009200 8618 raft_consensus.cc:740] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8324d6c954fa44908364578b4cd81a3f, State: Initialized, Role: FOLLOWER
I20260318 09:00:08.010138 8618 consensus_queue.cc:260] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf17c8d0b3e34fd28d632d53f9d7153b" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 35139 } } peers { permanent_uuid: "a0c912cb1f164f3c99ba56f923ac350f" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 35601 } } peers { permanent_uuid: "8324d6c954fa44908364578b4cd81a3f" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33873 } }
I20260318 09:00:08.012892 8618 ts_tablet_manager.cc:1434] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260318 09:00:08.013233 8618 ts_tablet_manager.cc:1457] Injecting 50ms of latency into OpenTablet
I20260318 09:00:08.017349 8621 ts_tablet_manager.cc:1929] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260318 09:00:08.017841 8621 log.cc:1199] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: Deleting WAL directory at /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-1-root/wals/c6dcb4f32a544b738afa5da77805c16a
I20260318 09:00:08.018714 8621 ts_tablet_manager.cc:1950] T c6dcb4f32a544b738afa5da77805c16a P a0c912cb1f164f3c99ba56f923ac350f: Deleting consensus metadata
I20260318 09:00:08.018936 8404 tablet_service.cc:1558] Processing DeleteTablet for tablet c6dcb4f32a544b738afa5da77805c16a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-18 09:00:07 UTC) from {username='slave'} at 127.0.0.1:43192
I20260318 09:00:08.019871 8622 ts_tablet_manager.cc:1929] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260318 09:00:08.020234 8622 log.cc:1199] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: Deleting WAL directory at /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-2-root/wals/c6dcb4f32a544b738afa5da77805c16a
W20260318 09:00:08.020509 8297 catalog_manager.cc:4982] TS 8324d6c954fa44908364578b4cd81a3f (127.6.2.65:33873): delete failed for tablet c6dcb4f32a544b738afa5da77805c16a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6dcb4f32a544b738afa5da77805c16a already in progress: copying tablet
I20260318 09:00:08.021055 8622 ts_tablet_manager.cc:1950] T c6dcb4f32a544b738afa5da77805c16a P cf17c8d0b3e34fd28d632d53f9d7153b: Deleting consensus metadata
I20260318 09:00:08.021668 8297 catalog_manager.cc:5002] TS a0c912cb1f164f3c99ba56f923ac350f (127.6.2.66:35601): tablet c6dcb4f32a544b738afa5da77805c16a (table test-table [id=a72231a9928e4591a75143a413d0b034]) successfully deleted
I20260318 09:00:08.022485 8299 catalog_manager.cc:5002] TS cf17c8d0b3e34fd28d632d53f9d7153b (127.6.2.67:35139): tablet c6dcb4f32a544b738afa5da77805c16a (table test-table [id=a72231a9928e4591a75143a413d0b034]) successfully deleted
I20260318 09:00:08.065323 8573 tablet_copy_service.cc:342] P cf17c8d0b3e34fd28d632d53f9d7153b: Request end of tablet copy session 8324d6c954fa44908364578b4cd81a3f-c6dcb4f32a544b738afa5da77805c16a received from {username='slave'} at 127.0.0.1:56344
I20260318 09:00:08.065814 8573 tablet_copy_service.cc:434] P cf17c8d0b3e34fd28d632d53f9d7153b: ending tablet copy session 8324d6c954fa44908364578b4cd81a3f-c6dcb4f32a544b738afa5da77805c16a on tablet c6dcb4f32a544b738afa5da77805c16a with peer 8324d6c954fa44908364578b4cd81a3f
I20260318 09:00:08.070354 8404 tablet_service.cc:1558] Processing DeleteTablet for tablet c6dcb4f32a544b738afa5da77805c16a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-18 09:00:07 UTC) from {username='slave'} at 127.0.0.1:43192
I20260318 09:00:08.071349 8617 tablet_replica.cc:333] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: stopping tablet replica
I20260318 09:00:08.071970 8617 raft_consensus.cc:2243] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:08.072350 8617 pending_rounds.cc:70] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Trying to abort 1 pending ops.
I20260318 09:00:08.072589 8617 pending_rounds.cc:77] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265584774533484544 op_type: NO_OP noop_request { }
I20260318 09:00:08.072909 8617 raft_consensus.cc:2889] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260318 09:00:08.073242 8617 raft_consensus.cc:2272] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:08.075302 8617 ts_tablet_manager.cc:1916] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260318 09:00:08.085712 8617 ts_tablet_manager.cc:1929] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260318 09:00:08.086103 8617 log.cc:1199] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Deleting WAL directory at /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1773824355427976-6153-0/minicluster-data/ts-0-root/wals/c6dcb4f32a544b738afa5da77805c16a
I20260318 09:00:08.086723 8617 ts_tablet_manager.cc:1950] T c6dcb4f32a544b738afa5da77805c16a P 8324d6c954fa44908364578b4cd81a3f: Deleting consensus metadata
I20260318 09:00:08.089157 8297 catalog_manager.cc:5002] TS 8324d6c954fa44908364578b4cd81a3f (127.6.2.65:33873): tablet c6dcb4f32a544b738afa5da77805c16a (table test-table [id=a72231a9928e4591a75143a413d0b034]) successfully deleted
I20260318 09:00:08.131290 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 09:00:08.152632 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:08.160714 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 09:00:08.169757 6153 tablet_server.cc:178] TabletServer@127.6.2.66:0 shutting down...
I20260318 09:00:08.188359 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:08.206321 6153 tablet_server.cc:195] TabletServer@127.6.2.66:0 shutdown complete.
I20260318 09:00:08.217854 6153 tablet_server.cc:178] TabletServer@127.6.2.67:0 shutting down...
I20260318 09:00:08.242576 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:08.250082 6153 tablet_server.cc:195] TabletServer@127.6.2.67:0 shutdown complete.
W20260318 09:00:08.251930 8343 debug-util.cc:398] Leaking SignalData structure 0x7b08000affc0 after lost signal to thread 6156
I20260318 09:00:08.262423 6153 master.cc:561] Master@127.6.2.126:38655 shutting down...
I20260318 09:00:08.283737 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:08.284365 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:08.284862 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8eb6b42bb0814254a45b17c07d9d2f1e: stopping tablet replica
W20260318 09:00:09.292518 6153 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 8343)
W20260318 09:00:10.293015 6153 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 8343)
I20260318 09:00:10.728554 6153 master.cc:583] Master@127.6.2.126:38655 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (3899 ms)
[----------] 4 tests from TsTabletManagerITest (45189 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20260318 09:00:10.755944 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:39381
I20260318 09:00:10.757103 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:10.762449 8628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:10.763743 8629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:10.765625 8631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:10.766481 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:10.768863 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:10.769131 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:10.769330 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824410769313 us; error 0 us; skew 500 ppm
I20260318 09:00:10.769920 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:10.772543 6153 webserver.cc:492] Webserver started at http://127.6.2.126:33089/ using document root <none> and password file <none>
I20260318 09:00:10.773061 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:10.773262 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:10.773576 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:10.774698 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "74f28ce0e79141c2b91eebd51575fc7f"
format_stamp: "Formatted at 2026-03-18 09:00:10 on dist-test-slave-mpvv"
I20260318 09:00:10.779395 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260318 09:00:10.782939 8636 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:10.783833 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260318 09:00:10.784165 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "74f28ce0e79141c2b91eebd51575fc7f"
format_stamp: "Formatted at 2026-03-18 09:00:10 on dist-test-slave-mpvv"
I20260318 09:00:10.784499 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:10.798156 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:10.799332 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:10.836318 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:39381
I20260318 09:00:10.836416 8687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:39381 every 8 connection(s)
I20260318 09:00:10.840283 8688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:10.850831 8688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f: Bootstrap starting.
I20260318 09:00:10.855767 8688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:10.859758 8688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f: No bootstrap required, opened a new log
I20260318 09:00:10.861660 8688 raft_consensus.cc:359] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f28ce0e79141c2b91eebd51575fc7f" member_type: VOTER }
I20260318 09:00:10.862036 8688 raft_consensus.cc:385] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:10.862216 8688 raft_consensus.cc:740] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f28ce0e79141c2b91eebd51575fc7f, State: Initialized, Role: FOLLOWER
I20260318 09:00:10.862787 8688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f28ce0e79141c2b91eebd51575fc7f" member_type: VOTER }
I20260318 09:00:10.863289 8688 raft_consensus.cc:399] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:10.863504 8688 raft_consensus.cc:493] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:10.863724 8688 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:10.868120 8688 raft_consensus.cc:515] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f28ce0e79141c2b91eebd51575fc7f" member_type: VOTER }
I20260318 09:00:10.868620 8688 leader_election.cc:304] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f28ce0e79141c2b91eebd51575fc7f; no voters:
I20260318 09:00:10.869848 8688 leader_election.cc:290] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:10.870227 8691 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:10.871641 8691 raft_consensus.cc:697] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 1 LEADER]: Becoming Leader. State: Replica: 74f28ce0e79141c2b91eebd51575fc7f, State: Running, Role: LEADER
I20260318 09:00:10.872347 8691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f28ce0e79141c2b91eebd51575fc7f" member_type: VOTER }
I20260318 09:00:10.873034 8688 sys_catalog.cc:565] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [sys.catalog]: configured and running, proceeding with master startup.
I20260318 09:00:10.877506 8692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f28ce0e79141c2b91eebd51575fc7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f28ce0e79141c2b91eebd51575fc7f" member_type: VOTER } }
I20260318 09:00:10.878263 8692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:10.879817 8693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f28ce0e79141c2b91eebd51575fc7f. Latest consensus state: current_term: 1 leader_uuid: "74f28ce0e79141c2b91eebd51575fc7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f28ce0e79141c2b91eebd51575fc7f" member_type: VOTER } }
I20260318 09:00:10.880594 8698 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 09:00:10.880652 8693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:10.885533 8698 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 09:00:10.889984 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 09:00:10.894908 8698 catalog_manager.cc:1357] Generated new cluster ID: f44cfe26cbf644d6be54926d04c1a8e7
I20260318 09:00:10.895221 8698 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 09:00:10.907735 8698 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 09:00:10.909057 8698 catalog_manager.cc:1514] Loading token signing keys...
I20260318 09:00:10.924950 8698 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f: Generated new TSK 0
I20260318 09:00:10.925602 8698 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 09:00:10.956391 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:10.962473 8709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:10.963513 8710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:10.965444 6153 server_base.cc:1060] running on GCE node
W20260318 09:00:10.965859 8712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:10.966776 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:10.967026 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:10.967242 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824410967218 us; error 0 us; skew 500 ppm
I20260318 09:00:10.967811 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:10.970223 6153 webserver.cc:492] Webserver started at http://127.6.2.65:40883/ using document root <none> and password file <none>
I20260318 09:00:10.970742 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:10.970968 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:10.971328 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:10.972421 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "203e43b22bc34b30aebb6696e6175ebd"
format_stamp: "Formatted at 2026-03-18 09:00:10 on dist-test-slave-mpvv"
I20260318 09:00:10.977248 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260318 09:00:10.980731 8717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:10.981703 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260318 09:00:10.981992 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "203e43b22bc34b30aebb6696e6175ebd"
format_stamp: "Formatted at 2026-03-18 09:00:10 on dist-test-slave-mpvv"
I20260318 09:00:10.982240 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:10.994421 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:10.995656 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:11.001055 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:11.001456 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:11.001861 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:11.002148 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:11.058238 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:33701
I20260318 09:00:11.058339 8787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:33701 every 8 connection(s)
I20260318 09:00:11.062954 8788 heartbeater.cc:344] Connected to a master server at 127.6.2.126:39381
I20260318 09:00:11.063390 8788 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:11.064257 8788 heartbeater.cc:507] Master 127.6.2.126:39381 requested a full tablet report, sending...
I20260318 09:00:11.066345 8653 ts_manager.cc:194] Registered new tserver with Master: 203e43b22bc34b30aebb6696e6175ebd (127.6.2.65:33701)
I20260318 09:00:11.066470 6153 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004983017s
I20260318 09:00:11.068228 8653 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47494
I20260318 09:00:11.100958 8753 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260318 09:00:11.102288 8753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:11.114667 8798 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: Bootstrap starting.
I20260318 09:00:11.119400 8798 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:11.123574 8798 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: No bootstrap required, opened a new log
I20260318 09:00:11.123996 8798 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260318 09:00:11.125924 8798 raft_consensus.cc:359] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } }
I20260318 09:00:11.126307 8798 raft_consensus.cc:385] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:11.126583 8798 raft_consensus.cc:740] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203e43b22bc34b30aebb6696e6175ebd, State: Initialized, Role: FOLLOWER
I20260318 09:00:11.127194 8798 consensus_queue.cc:260] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } }
I20260318 09:00:11.127676 8798 raft_consensus.cc:399] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:11.127909 8798 raft_consensus.cc:493] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:11.128203 8798 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:11.132915 8798 raft_consensus.cc:515] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } }
I20260318 09:00:11.133563 8798 leader_election.cc:304] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 203e43b22bc34b30aebb6696e6175ebd; no voters:
I20260318 09:00:11.134768 8798 leader_election.cc:290] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:11.135222 8800 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:11.136639 8800 raft_consensus.cc:697] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 1 LEADER]: Becoming Leader. State: Replica: 203e43b22bc34b30aebb6696e6175ebd, State: Running, Role: LEADER
I20260318 09:00:11.136865 8788 heartbeater.cc:499] Master 127.6.2.126:39381 was elected leader, sending a full tablet report...
I20260318 09:00:11.137512 8798 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260318 09:00:11.137493 8800 consensus_queue.cc:237] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } }
I20260318 09:00:11.142179 8802 tablet_replica.cc:442] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: TxnStatusTablet state changed. Reason: New leader 203e43b22bc34b30aebb6696e6175ebd. Latest consensus state: current_term: 1 leader_uuid: "203e43b22bc34b30aebb6696e6175ebd" committed_config { opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } } }
I20260318 09:00:11.142390 8801 tablet_replica.cc:442] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "203e43b22bc34b30aebb6696e6175ebd" committed_config { opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } } }
I20260318 09:00:11.142989 8802 tablet_replica.cc:445] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: This TxnStatusTablet replica's current role is: LEADER
I20260318 09:00:11.143067 8801 tablet_replica.cc:445] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: This TxnStatusTablet replica's current role is: LEADER
W20260318 09:00:11.145265 8653 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:11.146148 8805 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260318 09:00:11.146754 8805 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260318 09:00:11.150184 8653 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:11.181576 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 09:00:11.205300 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:11.206197 6153 tablet_replica.cc:333] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: stopping tablet replica
I20260318 09:00:11.206861 6153 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:11.207391 6153 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:11.229858 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 09:00:11.241248 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:11.247380 8808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:11.249568 8809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:11.250499 8811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:11.251700 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:11.252889 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:11.253366 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:11.253603 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824411253577 us; error 0 us; skew 500 ppm
I20260318 09:00:11.254158 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:11.257303 6153 webserver.cc:492] Webserver started at http://127.6.2.65:40883/ using document root <none> and password file <none>
I20260318 09:00:11.257848 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:11.258064 6153 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260318 09:00:11.262054 6153 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260318 09:00:11.265353 8816 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:11.266371 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260318 09:00:11.266754 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "203e43b22bc34b30aebb6696e6175ebd"
format_stamp: "Formatted at 2026-03-18 09:00:10 on dist-test-slave-mpvv"
I20260318 09:00:11.267132 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:11.292688 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:11.294131 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:11.303092 8825 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20260318 09:00:11.312191 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260318 09:00:11.312497 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s user 0.001s sys 0.001s
I20260318 09:00:11.312819 6153 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260318 09:00:11.317531 6153 ts_tablet_manager.cc:616] Registered 1 tablets
I20260318 09:00:11.317849 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.005s user 0.005s sys 0.001s
I20260318 09:00:11.317899 8825 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: Bootstrap starting.
I20260318 09:00:11.362135 8825 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260318 09:00:11.363046 8825 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: Bootstrap complete.
I20260318 09:00:11.363886 8825 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: Time spent bootstrapping tablet: real 0.046s user 0.027s sys 0.014s
I20260318 09:00:11.365842 8825 raft_consensus.cc:359] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } }
I20260318 09:00:11.366313 8825 raft_consensus.cc:740] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203e43b22bc34b30aebb6696e6175ebd, State: Initialized, Role: FOLLOWER
I20260318 09:00:11.367341 8825 consensus_queue.cc:260] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } }
I20260318 09:00:11.368111 8825 raft_consensus.cc:399] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:11.368527 8825 raft_consensus.cc:493] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:11.368930 8825 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 1 FOLLOWER]: Advancing to term 2
I20260318 09:00:11.377173 8825 raft_consensus.cc:515] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } }
I20260318 09:00:11.378355 8825 leader_election.cc:304] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 203e43b22bc34b30aebb6696e6175ebd; no voters:
I20260318 09:00:11.386029 8825 leader_election.cc:290] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260318 09:00:11.387460 8852 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 2 FOLLOWER]: Leader election won for term 2
I20260318 09:00:11.395694 8825 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd: Time spent starting tablet: real 0.031s user 0.021s sys 0.004s
I20260318 09:00:11.397909 8852 raft_consensus.cc:697] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 2 LEADER]: Becoming Leader. State: Replica: 203e43b22bc34b30aebb6696e6175ebd, State: Running, Role: LEADER
I20260318 09:00:11.398823 8852 consensus_queue.cc:237] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } }
I20260318 09:00:11.402158 8860 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "203e43b22bc34b30aebb6696e6175ebd" committed_config { opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } } }
I20260318 09:00:11.402815 8860 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260318 09:00:11.403718 8863 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 203e43b22bc34b30aebb6696e6175ebd. Latest consensus state: current_term: 2 leader_uuid: "203e43b22bc34b30aebb6696e6175ebd" committed_config { opid_index: -1 peers { permanent_uuid: "203e43b22bc34b30aebb6696e6175ebd" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33701 } } }
I20260318 09:00:11.405903 8863 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260318 09:00:11.412753 8881 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260318 09:00:11.413273 8881 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260318 09:00:11.418632 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:33701
I20260318 09:00:11.418793 8893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:33701 every 8 connection(s)
I20260318 09:00:11.426347 8894 heartbeater.cc:344] Connected to a master server at 127.6.2.126:39381
I20260318 09:00:11.426929 8894 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:11.427901 8894 heartbeater.cc:507] Master 127.6.2.126:39381 requested a full tablet report, sending...
I20260318 09:00:11.430974 6153 tablet_server.cc:178] TabletServer@127.6.2.65:33701 shutting down...
I20260318 09:00:11.431504 8653 ts_manager.cc:194] Re-registered known tserver with Master: 203e43b22bc34b30aebb6696e6175ebd (127.6.2.65:33701)
W20260318 09:00:11.432113 8653 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:11.433907 8653 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47532
I20260318 09:00:11.438352 8894 heartbeater.cc:499] Master 127.6.2.126:39381 was elected leader, sending a full tablet report...
W20260318 09:00:11.441690 8653 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:11.461247 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:11.462193 6153 tablet_replica.cc:333] stopping tablet replica
I20260318 09:00:11.462852 6153 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 2 LEADER]: Raft consensus shutting down.
I20260318 09:00:11.463435 6153 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 203e43b22bc34b30aebb6696e6175ebd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:11.485327 6153 tablet_server.cc:195] TabletServer@127.6.2.65:33701 shutdown complete.
I20260318 09:00:11.498872 6153 master.cc:561] Master@127.6.2.126:39381 shutting down...
I20260318 09:00:11.516394 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:11.516996 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:11.517364 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 74f28ce0e79141c2b91eebd51575fc7f: stopping tablet replica
I20260318 09:00:11.536294 6153 master.cc:583] Master@127.6.2.126:39381 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (805 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20260318 09:00:11.561861 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:40371
I20260318 09:00:11.562956 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:11.568549 8900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:11.576160 8901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:11.578831 8903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:11.578989 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:11.579928 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:11.580118 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:11.580303 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824411580289 us; error 0 us; skew 500 ppm
I20260318 09:00:11.580922 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:11.583562 6153 webserver.cc:492] Webserver started at http://127.6.2.126:45599/ using document root <none> and password file <none>
I20260318 09:00:11.584095 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:11.584300 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:11.584582 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:11.585678 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "f5f14114ebba4cbcae4adc3bf6964f2a"
format_stamp: "Formatted at 2026-03-18 09:00:11 on dist-test-slave-mpvv"
I20260318 09:00:11.590605 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260318 09:00:11.595403 8908 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:11.596783 6153 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260318 09:00:11.597121 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "f5f14114ebba4cbcae4adc3bf6964f2a"
format_stamp: "Formatted at 2026-03-18 09:00:11 on dist-test-slave-mpvv"
I20260318 09:00:11.597471 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:11.609535 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:11.610608 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:11.664078 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:40371
I20260318 09:00:11.664170 8959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:40371 every 8 connection(s)
I20260318 09:00:11.668838 8960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:11.680079 8960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a: Bootstrap starting.
I20260318 09:00:11.684770 8960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:11.689311 8960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a: No bootstrap required, opened a new log
I20260318 09:00:11.691565 8960 raft_consensus.cc:359] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f14114ebba4cbcae4adc3bf6964f2a" member_type: VOTER }
I20260318 09:00:11.691968 8960 raft_consensus.cc:385] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:11.692238 8960 raft_consensus.cc:740] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f14114ebba4cbcae4adc3bf6964f2a, State: Initialized, Role: FOLLOWER
I20260318 09:00:11.692920 8960 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f14114ebba4cbcae4adc3bf6964f2a" member_type: VOTER }
I20260318 09:00:11.693427 8960 raft_consensus.cc:399] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:11.693656 8960 raft_consensus.cc:493] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:11.693976 8960 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:11.698851 8960 raft_consensus.cc:515] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f14114ebba4cbcae4adc3bf6964f2a" member_type: VOTER }
I20260318 09:00:11.699647 8960 leader_election.cc:304] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5f14114ebba4cbcae4adc3bf6964f2a; no voters:
I20260318 09:00:11.700872 8960 leader_election.cc:290] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:11.701260 8963 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:11.702912 8963 raft_consensus.cc:697] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 1 LEADER]: Becoming Leader. State: Replica: f5f14114ebba4cbcae4adc3bf6964f2a, State: Running, Role: LEADER
I20260318 09:00:11.703668 8963 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f14114ebba4cbcae4adc3bf6964f2a" member_type: VOTER }
I20260318 09:00:11.704135 8960 sys_catalog.cc:565] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [sys.catalog]: configured and running, proceeding with master startup.
I20260318 09:00:11.706595 8964 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5f14114ebba4cbcae4adc3bf6964f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f14114ebba4cbcae4adc3bf6964f2a" member_type: VOTER } }
I20260318 09:00:11.707136 8964 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:11.716347 8965 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5f14114ebba4cbcae4adc3bf6964f2a. Latest consensus state: current_term: 1 leader_uuid: "f5f14114ebba4cbcae4adc3bf6964f2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5f14114ebba4cbcae4adc3bf6964f2a" member_type: VOTER } }
I20260318 09:00:11.717051 8965 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:11.719470 8968 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 09:00:11.727424 8968 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 09:00:11.728572 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 09:00:11.737816 8968 catalog_manager.cc:1357] Generated new cluster ID: 0caf38e31619447b9863f28dfff67a0d
I20260318 09:00:11.738140 8968 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 09:00:11.754520 8968 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 09:00:11.755927 8968 catalog_manager.cc:1514] Loading token signing keys...
I20260318 09:00:11.772315 8968 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a: Generated new TSK 0
I20260318 09:00:11.772969 8968 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 09:00:11.795322 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:11.801842 8982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:11.803570 8983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:11.807897 8985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:11.808606 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:11.809664 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:11.809954 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:11.810150 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824411810135 us; error 0 us; skew 500 ppm
I20260318 09:00:11.810861 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:11.813865 6153 webserver.cc:492] Webserver started at http://127.6.2.65:46271/ using document root <none> and password file <none>
I20260318 09:00:11.814532 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:11.814777 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:11.815099 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:11.816629 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "74b314acd456428ebe34b743e0b7feb8"
format_stamp: "Formatted at 2026-03-18 09:00:11 on dist-test-slave-mpvv"
I20260318 09:00:11.822945 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260318 09:00:11.827728 8990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:11.828845 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260318 09:00:11.829217 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "74b314acd456428ebe34b743e0b7feb8"
format_stamp: "Formatted at 2026-03-18 09:00:11 on dist-test-slave-mpvv"
I20260318 09:00:11.829589 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:11.857398 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:11.858884 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:11.871415 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:11.871744 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:11.872068 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:11.872277 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:11.946715 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:43257
I20260318 09:00:11.946810 9060 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:43257 every 8 connection(s)
I20260318 09:00:11.952683 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260318 09:00:11.952708 9061 heartbeater.cc:344] Connected to a master server at 127.6.2.126:40371
I20260318 09:00:11.953413 9061 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:11.954170 9061 heartbeater.cc:507] Master 127.6.2.126:40371 requested a full tablet report, sending...
I20260318 09:00:11.956485 8925 ts_manager.cc:194] Registered new tserver with Master: 74b314acd456428ebe34b743e0b7feb8 (127.6.2.65:43257)
I20260318 09:00:11.958676 8925 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47620
W20260318 09:00:11.960813 9063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:11.963734 9064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:11.965832 9066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:11.965906 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:11.967247 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:11.967516 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:11.967729 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824411967713 us; error 0 us; skew 500 ppm
I20260318 09:00:11.968474 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:11.971673 6153 webserver.cc:492] Webserver started at http://127.6.2.66:41441/ using document root <none> and password file <none>
I20260318 09:00:11.972590 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:11.972922 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:11.973284 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:11.974705 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-1-root/instance:
uuid: "64a9753da9464d01a5f3c98ff77632b5"
format_stamp: "Formatted at 2026-03-18 09:00:11 on dist-test-slave-mpvv"
I20260318 09:00:11.981101 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260318 09:00:11.986090 9071 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:11.987035 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260318 09:00:11.987463 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-1-root
uuid: "64a9753da9464d01a5f3c98ff77632b5"
format_stamp: "Formatted at 2026-03-18 09:00:11 on dist-test-slave-mpvv"
I20260318 09:00:11.987848 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:12.017969 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:12.019424 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:12.026502 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:12.026834 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:12.027128 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:12.027366 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260318 09:00:12.094436 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.66:40323
I20260318 09:00:12.094520 9140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.66:40323 every 8 connection(s)
I20260318 09:00:12.100220 9141 heartbeater.cc:344] Connected to a master server at 127.6.2.126:40371
I20260318 09:00:12.100674 9141 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:12.101550 9141 heartbeater.cc:507] Master 127.6.2.126:40371 requested a full tablet report, sending...
I20260318 09:00:12.103782 8925 ts_manager.cc:194] Registered new tserver with Master: 64a9753da9464d01a5f3c98ff77632b5 (127.6.2.66:40323)
I20260318 09:00:12.104100 6153 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.005180899s
I20260318 09:00:12.105684 8925 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47652
I20260318 09:00:12.140816 9026 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260318 09:00:12.142463 9026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:12.159706 9151 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: Bootstrap starting.
I20260318 09:00:12.164798 9151 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:12.169380 9151 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: No bootstrap required, opened a new log
I20260318 09:00:12.169831 9151 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260318 09:00:12.171974 9151 raft_consensus.cc:359] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } }
I20260318 09:00:12.172411 9151 raft_consensus.cc:385] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:12.172649 9151 raft_consensus.cc:740] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b314acd456428ebe34b743e0b7feb8, State: Initialized, Role: FOLLOWER
I20260318 09:00:12.173240 9151 consensus_queue.cc:260] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } }
I20260318 09:00:12.173732 9151 raft_consensus.cc:399] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:12.173974 9151 raft_consensus.cc:493] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:12.174242 9151 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:12.178944 9151 raft_consensus.cc:515] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } }
I20260318 09:00:12.179719 9151 leader_election.cc:304] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74b314acd456428ebe34b743e0b7feb8; no voters:
I20260318 09:00:12.181195 9151 leader_election.cc:290] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:12.181583 9153 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:12.183334 9061 heartbeater.cc:499] Master 127.6.2.126:40371 was elected leader, sending a full tablet report...
I20260318 09:00:12.184368 9151 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260318 09:00:12.184638 9153 raft_consensus.cc:697] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 1 LEADER]: Becoming Leader. State: Replica: 74b314acd456428ebe34b743e0b7feb8, State: Running, Role: LEADER
I20260318 09:00:12.185380 9153 consensus_queue.cc:237] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } }
I20260318 09:00:12.189682 9155 tablet_replica.cc:442] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: TxnStatusTablet state changed. Reason: New leader 74b314acd456428ebe34b743e0b7feb8. Latest consensus state: current_term: 1 leader_uuid: "74b314acd456428ebe34b743e0b7feb8" committed_config { opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } } }
I20260318 09:00:12.190284 9155 tablet_replica.cc:445] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: This TxnStatusTablet replica's current role is: LEADER
W20260318 09:00:12.192128 8925 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:12.194890 9154 tablet_replica.cc:442] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74b314acd456428ebe34b743e0b7feb8" committed_config { opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } } }
I20260318 09:00:12.195505 9154 tablet_replica.cc:445] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: This TxnStatusTablet replica's current role is: LEADER
I20260318 09:00:12.198632 9158 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260318 09:00:12.199121 9158 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260318 09:00:12.202996 8925 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:12.260917 9162 ts_tablet_manager.cc:933] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: Initiating tablet copy from peer 74b314acd456428ebe34b743e0b7feb8 (127.6.2.65:43257)
I20260318 09:00:12.262929 9162 tablet_copy_client.cc:323] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: tablet copy: Beginning tablet copy session from remote peer at address 127.6.2.65:43257
I20260318 09:00:12.273202 9046 tablet_copy_service.cc:140] P 74b314acd456428ebe34b743e0b7feb8: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 64a9753da9464d01a5f3c98ff77632b5 ({username='slave'} at 127.0.0.1:40146)
I20260318 09:00:12.273753 9046 tablet_copy_service.cc:161] P 74b314acd456428ebe34b743e0b7feb8: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 64a9753da9464d01a5f3c98ff77632b5 at {username='slave'} at 127.0.0.1:40146: session id = 64a9753da9464d01a5f3c98ff77632b5-11111111111111111111111111111111
I20260318 09:00:12.279305 9046 tablet_copy_source_session.cc:215] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: Tablet Copy: opened 0 blocks and 1 log segments
I20260318 09:00:12.282184 9162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:12.294099 9162 tablet_copy_client.cc:806] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: tablet copy: Starting download of 0 data blocks...
I20260318 09:00:12.294725 9162 tablet_copy_client.cc:670] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: tablet copy: Starting download of 1 WAL segments...
I20260318 09:00:12.298430 9162 tablet_copy_client.cc:538] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260318 09:00:12.305145 9162 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: Bootstrap starting.
I20260318 09:00:12.328308 9162 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260318 09:00:12.329126 9162 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: Bootstrap complete.
I20260318 09:00:12.329716 9162 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.004s
I20260318 09:00:12.331921 9162 raft_consensus.cc:359] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } }
I20260318 09:00:12.332368 9162 raft_consensus.cc:740] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 64a9753da9464d01a5f3c98ff77632b5, State: Initialized, Role: NON_PARTICIPANT
I20260318 09:00:12.332937 9162 consensus_queue.cc:260] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } }
I20260318 09:00:12.334712 9165 tablet_replica.cc:442] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } } }
I20260318 09:00:12.334898 9141 heartbeater.cc:499] Master 127.6.2.126:40371 was elected leader, sending a full tablet report...
I20260318 09:00:12.335265 9165 tablet_replica.cc:445] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20260318 09:00:12.335584 9162 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260318 09:00:12.337376 9046 tablet_copy_service.cc:342] P 74b314acd456428ebe34b743e0b7feb8: Request end of tablet copy session 64a9753da9464d01a5f3c98ff77632b5-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:40146
I20260318 09:00:12.337814 9046 tablet_copy_service.cc:434] P 74b314acd456428ebe34b743e0b7feb8: ending tablet copy session 64a9753da9464d01a5f3c98ff77632b5-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 64a9753da9464d01a5f3c98ff77632b5
W20260318 09:00:12.338024 8925 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260318 09:00:12.342649 8925 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:12.359508 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 09:00:12.385644 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:12.386794 6153 tablet_replica.cc:333] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: stopping tablet replica
I20260318 09:00:12.387470 6153 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:12.388043 6153 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:12.410508 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 09:00:12.421167 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:12.427453 9166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:12.428814 9167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:12.431386 6153 server_base.cc:1060] running on GCE node
W20260318 09:00:12.432601 9170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:12.433992 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:12.434377 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:12.434656 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824412434631 us; error 0 us; skew 500 ppm
I20260318 09:00:12.435734 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:12.439355 6153 webserver.cc:492] Webserver started at http://127.6.2.65:46271/ using document root <none> and password file <none>
I20260318 09:00:12.440254 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:12.440582 6153 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260318 09:00:12.445775 6153 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260318 09:00:12.449293 9174 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:12.450384 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260318 09:00:12.450747 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "74b314acd456428ebe34b743e0b7feb8"
format_stamp: "Formatted at 2026-03-18 09:00:11 on dist-test-slave-mpvv"
I20260318 09:00:12.451112 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:12.471997 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:12.473513 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:12.482183 9183 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20260318 09:00:12.489691 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260318 09:00:12.490123 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20260318 09:00:12.490523 6153 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260318 09:00:12.496963 6153 ts_tablet_manager.cc:616] Registered 1 tablets
I20260318 09:00:12.497321 9183 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: Bootstrap starting.
I20260318 09:00:12.497342 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.007s user 0.007s sys 0.000s
I20260318 09:00:12.534700 9183 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260318 09:00:12.535918 9183 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: Bootstrap complete.
I20260318 09:00:12.536619 9183 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: Time spent bootstrapping tablet: real 0.039s user 0.024s sys 0.011s
I20260318 09:00:12.539654 9183 raft_consensus.cc:359] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } }
I20260318 09:00:12.540315 9183 raft_consensus.cc:740] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b314acd456428ebe34b743e0b7feb8, State: Initialized, Role: FOLLOWER
I20260318 09:00:12.541054 9183 consensus_queue.cc:260] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } }
I20260318 09:00:12.541790 9183 raft_consensus.cc:399] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:12.542147 9183 raft_consensus.cc:493] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:12.542513 9183 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 1 FOLLOWER]: Advancing to term 2
I20260318 09:00:12.550215 9183 raft_consensus.cc:515] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } }
I20260318 09:00:12.551020 9183 leader_election.cc:304] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74b314acd456428ebe34b743e0b7feb8; no voters:
I20260318 09:00:12.552510 9183 leader_election.cc:290] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260318 09:00:12.553001 9215 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 2 FOLLOWER]: Leader election won for term 2
I20260318 09:00:12.565675 9183 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8: Time spent starting tablet: real 0.029s user 0.028s sys 0.001s
I20260318 09:00:12.568086 9215 raft_consensus.cc:697] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 2 LEADER]: Becoming Leader. State: Replica: 74b314acd456428ebe34b743e0b7feb8, State: Running, Role: LEADER
I20260318 09:00:12.569134 9215 consensus_queue.cc:237] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } }
I20260318 09:00:12.572623 9216 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74b314acd456428ebe34b743e0b7feb8" committed_config { opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } } }
I20260318 09:00:12.573190 9216 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260318 09:00:12.578783 9217 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 74b314acd456428ebe34b743e0b7feb8. Latest consensus state: current_term: 2 leader_uuid: "74b314acd456428ebe34b743e0b7feb8" committed_config { opid_index: -1 peers { permanent_uuid: "74b314acd456428ebe34b743e0b7feb8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 43257 } } }
I20260318 09:00:12.579588 9217 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20260318 09:00:12.584033 9224 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260318 09:00:12.584617 9224 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260318 09:00:12.606792 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:43257
I20260318 09:00:12.606860 9251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:43257 every 8 connection(s)
I20260318 09:00:12.613292 9252 heartbeater.cc:344] Connected to a master server at 127.6.2.126:40371
I20260318 09:00:12.613777 9252 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:12.614653 9252 heartbeater.cc:507] Master 127.6.2.126:40371 requested a full tablet report, sending...
I20260318 09:00:12.617065 6153 tablet_server.cc:178] TabletServer@127.6.2.65:43257 shutting down...
I20260318 09:00:12.618386 8925 ts_manager.cc:194] Re-registered known tserver with Master: 74b314acd456428ebe34b743e0b7feb8 (127.6.2.65:43257)
W20260318 09:00:12.619103 8925 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:12.620952 8925 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47668
I20260318 09:00:12.624190 9252 heartbeater.cc:499] Master 127.6.2.126:40371 was elected leader, sending a full tablet report...
W20260318 09:00:12.628182 8925 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:12.646669 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:12.647568 6153 tablet_replica.cc:333] stopping tablet replica
I20260318 09:00:12.648283 6153 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 2 LEADER]: Raft consensus shutting down.
I20260318 09:00:12.648846 6153 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 74b314acd456428ebe34b743e0b7feb8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:12.672016 6153 tablet_server.cc:195] TabletServer@127.6.2.65:43257 shutdown complete.
I20260318 09:00:12.682797 6153 tablet_server.cc:178] TabletServer@127.6.2.66:0 shutting down...
I20260318 09:00:12.707764 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:12.708995 6153 tablet_replica.cc:333] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5: stopping tablet replica
I20260318 09:00:12.709537 6153 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260318 09:00:12.709959 6153 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 64a9753da9464d01a5f3c98ff77632b5 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260318 09:00:12.729611 6153 tablet_server.cc:195] TabletServer@127.6.2.66:0 shutdown complete.
I20260318 09:00:12.740798 6153 master.cc:561] Master@127.6.2.126:40371 shutting down...
I20260318 09:00:12.758226 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:12.758781 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:12.759112 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P f5f14114ebba4cbcae4adc3bf6964f2a: stopping tablet replica
I20260318 09:00:12.778985 6153 master.cc:583] Master@127.6.2.126:40371 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1249 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20260318 09:00:12.813066 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:42305
I20260318 09:00:12.814496 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:12.820838 9258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:12.823189 9259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:12.824326 6153 server_base.cc:1060] running on GCE node
W20260318 09:00:12.825354 9261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:12.826277 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:12.826539 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:12.826730 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824412826713 us; error 0 us; skew 500 ppm
I20260318 09:00:12.827476 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:12.830451 6153 webserver.cc:492] Webserver started at http://127.6.2.126:33709/ using document root <none> and password file <none>
I20260318 09:00:12.831082 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:12.831375 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:12.831717 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:12.833161 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "5d11957cf7e34652adb108846a79ec7c"
format_stamp: "Formatted at 2026-03-18 09:00:12 on dist-test-slave-mpvv"
I20260318 09:00:12.839342 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260318 09:00:12.843057 9266 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:12.843981 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260318 09:00:12.844331 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "5d11957cf7e34652adb108846a79ec7c"
format_stamp: "Formatted at 2026-03-18 09:00:12 on dist-test-slave-mpvv"
I20260318 09:00:12.844686 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:12.863641 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:12.864671 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:12.915422 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:42305
I20260318 09:00:12.915521 9317 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:42305 every 8 connection(s)
I20260318 09:00:12.919981 9318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:12.935046 9318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c: Bootstrap starting.
I20260318 09:00:12.941354 9318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:12.946453 9318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c: No bootstrap required, opened a new log
I20260318 09:00:12.949170 9318 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d11957cf7e34652adb108846a79ec7c" member_type: VOTER }
I20260318 09:00:12.949694 9318 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:12.949976 9318 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d11957cf7e34652adb108846a79ec7c, State: Initialized, Role: FOLLOWER
I20260318 09:00:12.950600 9318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d11957cf7e34652adb108846a79ec7c" member_type: VOTER }
I20260318 09:00:12.951218 9318 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:12.951547 9318 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:12.951874 9318 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:12.958720 9318 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d11957cf7e34652adb108846a79ec7c" member_type: VOTER }
I20260318 09:00:12.959440 9318 leader_election.cc:304] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d11957cf7e34652adb108846a79ec7c; no voters:
I20260318 09:00:12.960919 9318 leader_election.cc:290] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:12.961313 9321 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:12.963258 9321 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 1 LEADER]: Becoming Leader. State: Replica: 5d11957cf7e34652adb108846a79ec7c, State: Running, Role: LEADER
I20260318 09:00:12.964016 9321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d11957cf7e34652adb108846a79ec7c" member_type: VOTER }
I20260318 09:00:12.964460 9318 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [sys.catalog]: configured and running, proceeding with master startup.
I20260318 09:00:12.966512 9322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d11957cf7e34652adb108846a79ec7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d11957cf7e34652adb108846a79ec7c" member_type: VOTER } }
I20260318 09:00:12.966965 9322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:12.967775 9323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d11957cf7e34652adb108846a79ec7c. Latest consensus state: current_term: 1 leader_uuid: "5d11957cf7e34652adb108846a79ec7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d11957cf7e34652adb108846a79ec7c" member_type: VOTER } }
I20260318 09:00:12.968381 9323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:12.982334 9326 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 09:00:12.989154 9326 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 09:00:12.996001 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 09:00:13.001389 9326 catalog_manager.cc:1357] Generated new cluster ID: bb84da296ce44080a01d48442574b0ca
I20260318 09:00:13.001739 9326 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 09:00:13.025914 9326 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 09:00:13.027235 9326 catalog_manager.cc:1514] Loading token signing keys...
I20260318 09:00:13.045644 9326 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c: Generated new TSK 0
I20260318 09:00:13.046365 9326 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 09:00:13.068398 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:13.075513 9339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:13.076588 9340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:13.078437 9342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:13.079303 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:13.080183 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:13.080425 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:13.080626 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824413080592 us; error 0 us; skew 500 ppm
I20260318 09:00:13.081152 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:13.084151 6153 webserver.cc:492] Webserver started at http://127.6.2.65:39903/ using document root <none> and password file <none>
I20260318 09:00:13.084640 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:13.084858 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:13.085142 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:13.086213 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "342ff9e107f34423a609122c3c330660"
format_stamp: "Formatted at 2026-03-18 09:00:13 on dist-test-slave-mpvv"
I20260318 09:00:13.091507 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260318 09:00:13.095234 9347 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:13.096195 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260318 09:00:13.096520 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "342ff9e107f34423a609122c3c330660"
format_stamp: "Formatted at 2026-03-18 09:00:13 on dist-test-slave-mpvv"
I20260318 09:00:13.096827 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:13.119051 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:13.120201 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:13.132707 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:13.133138 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:13.133536 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:13.133818 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:13.201370 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:33289
I20260318 09:00:13.201432 9417 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:33289 every 8 connection(s)
I20260318 09:00:13.207545 9418 heartbeater.cc:344] Connected to a master server at 127.6.2.126:42305
I20260318 09:00:13.207979 9418 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:13.208866 9418 heartbeater.cc:507] Master 127.6.2.126:42305 requested a full tablet report, sending...
I20260318 09:00:13.211019 9283 ts_manager.cc:194] Registered new tserver with Master: 342ff9e107f34423a609122c3c330660 (127.6.2.65:33289)
I20260318 09:00:13.211848 6153 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.006105045s
I20260318 09:00:13.213279 9283 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39018
I20260318 09:00:13.246546 9383 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260318 09:00:13.247905 9383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:13.259747 9428 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660: Bootstrap starting.
I20260318 09:00:13.264070 9428 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:13.268306 9428 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660: No bootstrap required, opened a new log
I20260318 09:00:13.268723 9428 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260318 09:00:13.270612 9428 raft_consensus.cc:359] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "342ff9e107f34423a609122c3c330660" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33289 } }
I20260318 09:00:13.271047 9428 raft_consensus.cc:385] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:13.271375 9428 raft_consensus.cc:740] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342ff9e107f34423a609122c3c330660, State: Initialized, Role: FOLLOWER
I20260318 09:00:13.271915 9428 consensus_queue.cc:260] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "342ff9e107f34423a609122c3c330660" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33289 } }
I20260318 09:00:13.272420 9428 raft_consensus.cc:399] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:13.272679 9428 raft_consensus.cc:493] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:13.272953 9428 raft_consensus.cc:3060] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:13.277462 9428 raft_consensus.cc:515] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "342ff9e107f34423a609122c3c330660" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33289 } }
I20260318 09:00:13.278030 9428 leader_election.cc:304] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 342ff9e107f34423a609122c3c330660; no voters:
I20260318 09:00:13.279486 9428 leader_election.cc:290] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:13.279857 9430 raft_consensus.cc:2804] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:13.281412 9418 heartbeater.cc:499] Master 127.6.2.126:42305 was elected leader, sending a full tablet report...
I20260318 09:00:13.282609 9428 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260318 09:00:13.282687 9430 raft_consensus.cc:697] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 1 LEADER]: Becoming Leader. State: Replica: 342ff9e107f34423a609122c3c330660, State: Running, Role: LEADER
I20260318 09:00:13.283531 9430 consensus_queue.cc:237] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "342ff9e107f34423a609122c3c330660" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33289 } }
I20260318 09:00:13.287091 9432 tablet_replica.cc:442] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660: TxnStatusTablet state changed. Reason: New leader 342ff9e107f34423a609122c3c330660. Latest consensus state: current_term: 1 leader_uuid: "342ff9e107f34423a609122c3c330660" committed_config { opid_index: -1 peers { permanent_uuid: "342ff9e107f34423a609122c3c330660" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33289 } } }
I20260318 09:00:13.287241 9431 tablet_replica.cc:442] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "342ff9e107f34423a609122c3c330660" committed_config { opid_index: -1 peers { permanent_uuid: "342ff9e107f34423a609122c3c330660" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 33289 } } }
I20260318 09:00:13.287658 9432 tablet_replica.cc:445] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660: This TxnStatusTablet replica's current role is: LEADER
I20260318 09:00:13.287819 9431 tablet_replica.cc:445] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660: This TxnStatusTablet replica's current role is: LEADER
W20260318 09:00:13.291863 9283 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:13.298213 9434 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260318 09:00:13.298698 9434 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260318 09:00:13.299520 9283 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260318 09:00:13.349584 9373 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260318 09:00:13.349931 9373 txn_status_manager.cc:244] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260318 09:00:13.350313 9373 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20260318 09:00:13.359055 9373 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260318 09:00:13.397914 9373 txn_status_manager.cc:244] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260318 09:00:13.398355 9373 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20260318 09:00:13.406149 9373 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20260318 09:00:13.417968 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:59570
W20260318 09:00:13.420478 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:59570
W20260318 09:00:13.422652 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:59570
W20260318 09:00:13.424693 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:59570
W20260318 09:00:13.426733 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:59570
W20260318 09:00:13.428871 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:59570
W20260318 09:00:13.439410 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:59578
W20260318 09:00:13.441630 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:59578
W20260318 09:00:13.443938 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:59578
W20260318 09:00:13.446086 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:59578
W20260318 09:00:13.448276 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:59578
W20260318 09:00:13.450193 9383 server_base.cc:1142] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:59578
I20260318 09:00:13.452968 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 09:00:13.482456 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:13.483541 6153 tablet_replica.cc:333] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660: stopping tablet replica
I20260318 09:00:13.484170 6153 raft_consensus.cc:2243] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:13.484673 6153 raft_consensus.cc:2272] T 11111111111111111111111111111111 P 342ff9e107f34423a609122c3c330660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:13.507277 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 09:00:13.519722 6153 master.cc:561] Master@127.6.2.126:42305 shutting down...
I20260318 09:00:13.548601 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:13.549250 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:13.549777 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 5d11957cf7e34652adb108846a79ec7c: stopping tablet replica
I20260318 09:00:13.568645 6153 master.cc:583] Master@127.6.2.126:42305 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (783 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20260318 09:00:13.595953 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:33003
I20260318 09:00:13.597128 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:13.602631 9443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:13.604197 9444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:13.606668 9446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:13.607302 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:13.608129 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:13.608304 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:13.608518 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824413608480 us; error 0 us; skew 500 ppm
I20260318 09:00:13.609061 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:13.612097 6153 webserver.cc:492] Webserver started at http://127.6.2.126:41327/ using document root <none> and password file <none>
I20260318 09:00:13.612589 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:13.612785 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:13.613067 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:13.614138 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "0ccca04c441047208afcca732ec29dd1"
format_stamp: "Formatted at 2026-03-18 09:00:13 on dist-test-slave-mpvv"
I20260318 09:00:13.619248 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260318 09:00:13.622853 9451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:13.623770 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260318 09:00:13.624111 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "0ccca04c441047208afcca732ec29dd1"
format_stamp: "Formatted at 2026-03-18 09:00:13 on dist-test-slave-mpvv"
I20260318 09:00:13.624464 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:13.659668 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:13.661043 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:13.713430 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:33003
I20260318 09:00:13.713517 9502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:33003 every 8 connection(s)
I20260318 09:00:13.718258 9503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:13.728586 9503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1: Bootstrap starting.
I20260318 09:00:13.732975 9503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:13.737079 9503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1: No bootstrap required, opened a new log
I20260318 09:00:13.739012 9503 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccca04c441047208afcca732ec29dd1" member_type: VOTER }
I20260318 09:00:13.739470 9503 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:13.739722 9503 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ccca04c441047208afcca732ec29dd1, State: Initialized, Role: FOLLOWER
I20260318 09:00:13.740356 9503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccca04c441047208afcca732ec29dd1" member_type: VOTER }
I20260318 09:00:13.740835 9503 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:13.741046 9503 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:13.741350 9503 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:13.745922 9503 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccca04c441047208afcca732ec29dd1" member_type: VOTER }
I20260318 09:00:13.746460 9503 leader_election.cc:304] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ccca04c441047208afcca732ec29dd1; no voters:
I20260318 09:00:13.747733 9503 leader_election.cc:290] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:13.748065 9506 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:13.749392 9506 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 1 LEADER]: Becoming Leader. State: Replica: 0ccca04c441047208afcca732ec29dd1, State: Running, Role: LEADER
I20260318 09:00:13.750056 9506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccca04c441047208afcca732ec29dd1" member_type: VOTER }
I20260318 09:00:13.750746 9503 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260318 09:00:13.754191 9507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ccca04c441047208afcca732ec29dd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccca04c441047208afcca732ec29dd1" member_type: VOTER } }
I20260318 09:00:13.754912 9507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:13.754853 9508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ccca04c441047208afcca732ec29dd1. Latest consensus state: current_term: 1 leader_uuid: "0ccca04c441047208afcca732ec29dd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccca04c441047208afcca732ec29dd1" member_type: VOTER } }
I20260318 09:00:13.755597 9508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:13.762837 9513 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 09:00:13.767328 9513 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 09:00:13.768575 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 09:00:13.776298 9513 catalog_manager.cc:1357] Generated new cluster ID: effa58fa4dbb48b2a4d13203297ab4cd
I20260318 09:00:13.776616 9513 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 09:00:13.789379 9513 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 09:00:13.790637 9513 catalog_manager.cc:1514] Loading token signing keys...
I20260318 09:00:13.812060 9513 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1: Generated new TSK 0
I20260318 09:00:13.812688 9513 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 09:00:13.835717 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:13.841634 9524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:13.843461 9525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:13.845209 9527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:13.845345 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:13.846371 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:13.846566 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:13.846803 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824413846750 us; error 0 us; skew 500 ppm
I20260318 09:00:13.847465 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:13.849858 6153 webserver.cc:492] Webserver started at http://127.6.2.65:41461/ using document root <none> and password file <none>
I20260318 09:00:13.850370 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:13.850572 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:13.850874 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:13.852126 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "e6bc835c38fa44c580db4478a9a497c8"
format_stamp: "Formatted at 2026-03-18 09:00:13 on dist-test-slave-mpvv"
I20260318 09:00:13.856710 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260318 09:00:13.860314 9532 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:13.861299 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260318 09:00:13.861608 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "e6bc835c38fa44c580db4478a9a497c8"
format_stamp: "Formatted at 2026-03-18 09:00:13 on dist-test-slave-mpvv"
I20260318 09:00:13.861871 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:13.874431 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:13.875496 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:13.881141 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:13.881465 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:13.881914 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:13.882153 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:13.933087 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:38291
I20260318 09:00:13.933202 9602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:38291 every 8 connection(s)
I20260318 09:00:13.939519 9603 heartbeater.cc:344] Connected to a master server at 127.6.2.126:33003
I20260318 09:00:13.939942 9603 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:13.940694 9603 heartbeater.cc:507] Master 127.6.2.126:33003 requested a full tablet report, sending...
I20260318 09:00:13.942750 9468 ts_manager.cc:194] Registered new tserver with Master: e6bc835c38fa44c580db4478a9a497c8 (127.6.2.65:38291)
I20260318 09:00:13.942878 6153 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004944411s
I20260318 09:00:13.944558 9468 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48362
I20260318 09:00:13.974218 9568 tablet_service.cc:1511] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260318 09:00:13.975977 9568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:13.989300 9613 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8: Bootstrap starting.
I20260318 09:00:13.994156 9613 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:13.998405 9613 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8: No bootstrap required, opened a new log
I20260318 09:00:13.998847 9613 ts_tablet_manager.cc:1403] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260318 09:00:14.000799 9613 raft_consensus.cc:359] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "e6bc835c38fa44c580db4478a9a497c8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 38291 } }
I20260318 09:00:14.001197 9613 raft_consensus.cc:385] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:14.001463 9613 raft_consensus.cc:740] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6bc835c38fa44c580db4478a9a497c8, State: Initialized, Role: FOLLOWER
I20260318 09:00:14.002087 9613 consensus_queue.cc:260] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e6bc835c38fa44c580db4478a9a497c8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 38291 } }
I20260318 09:00:14.002550 9613 raft_consensus.cc:399] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:14.002801 9613 raft_consensus.cc:493] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:14.003072 9613 raft_consensus.cc:3060] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:14.007766 9613 raft_consensus.cc:515] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "e6bc835c38fa44c580db4478a9a497c8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 38291 } }
I20260318 09:00:14.008355 9613 leader_election.cc:304] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6bc835c38fa44c580db4478a9a497c8; no voters:
I20260318 09:00:14.009652 9613 leader_election.cc:290] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:14.010027 9615 raft_consensus.cc:2804] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:14.011535 9615 raft_consensus.cc:697] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 1 LEADER]: Becoming Leader. State: Replica: e6bc835c38fa44c580db4478a9a497c8, State: Running, Role: LEADER
I20260318 09:00:14.011631 9603 heartbeater.cc:499] Master 127.6.2.126:33003 was elected leader, sending a full tablet report...
I20260318 09:00:14.012149 9613 ts_tablet_manager.cc:1434] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260318 09:00:14.012656 9615 consensus_queue.cc:237] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e6bc835c38fa44c580db4478a9a497c8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 38291 } }
I20260318 09:00:14.017563 9616 tablet_replica.cc:442] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6bc835c38fa44c580db4478a9a497c8" committed_config { opid_index: -1 peers { permanent_uuid: "e6bc835c38fa44c580db4478a9a497c8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 38291 } } }
I20260318 09:00:14.018215 9616 tablet_replica.cc:445] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8: This TxnStatusTablet replica's current role is: LEADER
I20260318 09:00:14.017944 9617 tablet_replica.cc:442] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8: TxnStatusTablet state changed. Reason: New leader e6bc835c38fa44c580db4478a9a497c8. Latest consensus state: current_term: 1 leader_uuid: "e6bc835c38fa44c580db4478a9a497c8" committed_config { opid_index: -1 peers { permanent_uuid: "e6bc835c38fa44c580db4478a9a497c8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 38291 } } }
I20260318 09:00:14.019136 9617 tablet_replica.cc:445] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8: This TxnStatusTablet replica's current role is: LEADER
W20260318 09:00:14.020280 9468 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:14.021999 9620 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260318 09:00:14.022420 9620 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20260318 09:00:14.024971 9468 catalog_manager.cc:5504] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260318 09:00:14.047502 9568 tablet_service.cc:1511] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260318 09:00:14.048703 9568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:14.059363 9613 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8: Bootstrap starting.
I20260318 09:00:14.064066 9613 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:14.068100 9613 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8: No bootstrap required, opened a new log
I20260318 09:00:14.068526 9613 ts_tablet_manager.cc:1403] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260318 09:00:14.070367 9613 raft_consensus.cc:359] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "e6bc835c38fa44c580db4478a9a497c8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 38291 } }
I20260318 09:00:14.070806 9613 raft_consensus.cc:385] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:14.071049 9613 raft_consensus.cc:740] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6bc835c38fa44c580db4478a9a497c8, State: Initialized, Role: FOLLOWER
I20260318 09:00:14.071658 9613 consensus_queue.cc:260] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e6bc835c38fa44c580db4478a9a497c8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 38291 } }
I20260318 09:00:14.072147 9613 raft_consensus.cc:399] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:14.072379 9613 raft_consensus.cc:493] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:14.072947 9613 raft_consensus.cc:3060] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:14.077649 9613 raft_consensus.cc:515] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "e6bc835c38fa44c580db4478a9a497c8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 38291 } }
I20260318 09:00:14.078183 9613 leader_election.cc:304] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6bc835c38fa44c580db4478a9a497c8; no voters:
I20260318 09:00:14.078724 9613 leader_election.cc:290] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:14.078895 9616 raft_consensus.cc:2804] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:14.079411 9616 raft_consensus.cc:697] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 1 LEADER]: Becoming Leader. State: Replica: e6bc835c38fa44c580db4478a9a497c8, State: Running, Role: LEADER
I20260318 09:00:14.080648 9613 ts_tablet_manager.cc:1434] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260318 09:00:14.080796 9616 consensus_queue.cc:237] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e6bc835c38fa44c580db4478a9a497c8" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 38291 } }
W20260318 09:00:14.086687 9468 catalog_manager.cc:5504] Ignoring report from unknown tablet 22222222222222222222222222222222
W20260318 09:00:14.090853 9468 catalog_manager.cc:5504] Ignoring report from unknown tablet 22222222222222222222222222222222
I20260318 09:00:14.108893 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 09:00:14.129015 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:14.129840 6153 tablet_replica.cc:333] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8: stopping tablet replica
I20260318 09:00:14.130421 6153 raft_consensus.cc:2243] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:14.130888 6153 raft_consensus.cc:2272] T 22222222222222222222222222222222 P e6bc835c38fa44c580db4478a9a497c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:14.133410 6153 tablet_replica.cc:333] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8: stopping tablet replica
I20260318 09:00:14.133942 6153 raft_consensus.cc:2243] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:14.134377 6153 raft_consensus.cc:2272] T 11111111111111111111111111111111 P e6bc835c38fa44c580db4478a9a497c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:14.155772 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 09:00:14.167935 6153 master.cc:561] Master@127.6.2.126:33003 shutting down...
I20260318 09:00:14.186308 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:14.186811 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:14.187243 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0ccca04c441047208afcca732ec29dd1: stopping tablet replica
I20260318 09:00:14.206787 6153 master.cc:583] Master@127.6.2.126:33003 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (636 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3475 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20260318 09:00:14.231611 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:45395
I20260318 09:00:14.232621 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:14.237726 9629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:14.238940 9630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:14.240795 9632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:14.241020 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:14.241840 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:14.242024 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:14.242157 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824414242139 us; error 0 us; skew 500 ppm
I20260318 09:00:14.242617 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:14.244961 6153 webserver.cc:492] Webserver started at http://127.6.2.126:40277/ using document root <none> and password file <none>
I20260318 09:00:14.245390 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:14.245553 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:14.245760 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:14.246778 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "1eb46555646e4a9b80c01034f074471c"
format_stamp: "Formatted at 2026-03-18 09:00:14 on dist-test-slave-mpvv"
I20260318 09:00:14.251219 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260318 09:00:14.254608 9637 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:14.255436 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260318 09:00:14.255712 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "1eb46555646e4a9b80c01034f074471c"
format_stamp: "Formatted at 2026-03-18 09:00:14 on dist-test-slave-mpvv"
I20260318 09:00:14.255962 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:14.266810 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:14.267861 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:14.304489 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:45395
I20260318 09:00:14.304605 9688 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:45395 every 8 connection(s)
I20260318 09:00:14.308475 9689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:14.319023 9689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c: Bootstrap starting.
I20260318 09:00:14.323559 9689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:14.327497 9689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c: No bootstrap required, opened a new log
I20260318 09:00:14.329581 9689 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb46555646e4a9b80c01034f074471c" member_type: VOTER }
I20260318 09:00:14.329963 9689 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:14.330152 9689 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb46555646e4a9b80c01034f074471c, State: Initialized, Role: FOLLOWER
I20260318 09:00:14.330608 9689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb46555646e4a9b80c01034f074471c" member_type: VOTER }
I20260318 09:00:14.331027 9689 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:14.331351 9689 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:14.331583 9689 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:14.336064 9689 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb46555646e4a9b80c01034f074471c" member_type: VOTER }
I20260318 09:00:14.336555 9689 leader_election.cc:304] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eb46555646e4a9b80c01034f074471c; no voters:
I20260318 09:00:14.337806 9689 leader_election.cc:290] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:14.338207 9692 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:14.339704 9692 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 1 LEADER]: Becoming Leader. State: Replica: 1eb46555646e4a9b80c01034f074471c, State: Running, Role: LEADER
I20260318 09:00:14.340363 9692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb46555646e4a9b80c01034f074471c" member_type: VOTER }
I20260318 09:00:14.341053 9689 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [sys.catalog]: configured and running, proceeding with master startup.
I20260318 09:00:14.343662 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eb46555646e4a9b80c01034f074471c. Latest consensus state: current_term: 1 leader_uuid: "1eb46555646e4a9b80c01034f074471c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb46555646e4a9b80c01034f074471c" member_type: VOTER } }
I20260318 09:00:14.343605 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eb46555646e4a9b80c01034f074471c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb46555646e4a9b80c01034f074471c" member_type: VOTER } }
I20260318 09:00:14.344393 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:14.344485 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:14.348743 9697 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 09:00:14.356294 9697 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 09:00:14.357327 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 09:00:14.365468 9697 catalog_manager.cc:1357] Generated new cluster ID: c1ba1c0dbbfe400796e967169333cfb0
I20260318 09:00:14.365828 9697 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 09:00:14.381028 9697 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 09:00:14.382331 9697 catalog_manager.cc:1514] Loading token signing keys...
I20260318 09:00:14.399260 9697 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c: Generated new TSK 0
I20260318 09:00:14.399945 9697 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 09:00:14.424031 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:14.429893 9710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:14.431430 9711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:14.432854 6153 server_base.cc:1060] running on GCE node
W20260318 09:00:14.432808 9713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:14.433959 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:14.434150 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:14.434366 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824414434329 us; error 0 us; skew 500 ppm
I20260318 09:00:14.434909 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:14.437224 6153 webserver.cc:492] Webserver started at http://127.6.2.65:38697/ using document root <none> and password file <none>
I20260318 09:00:14.437745 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:14.437956 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:14.438226 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:14.439427 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "9ef45de85edb48fd8a35599b4f764a35"
format_stamp: "Formatted at 2026-03-18 09:00:14 on dist-test-slave-mpvv"
I20260318 09:00:14.443851 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260318 09:00:14.447069 9718 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:14.447973 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260318 09:00:14.448321 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "9ef45de85edb48fd8a35599b4f764a35"
format_stamp: "Formatted at 2026-03-18 09:00:14 on dist-test-slave-mpvv"
I20260318 09:00:14.448617 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:14.471386 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:14.472405 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:14.473821 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 09:00:14.476192 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:14.476426 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:14.476691 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:14.476886 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:14.514981 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:35523
I20260318 09:00:14.515092 9780 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:35523 every 8 connection(s)
I20260318 09:00:14.519829 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:14.527081 9785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:14.528913 9786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:14.531607 6153 server_base.cc:1060] running on GCE node
W20260318 09:00:14.531607 9788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:14.532977 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:14.533252 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:14.533457 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824414533439 us; error 0 us; skew 500 ppm
I20260318 09:00:14.533855 9781 heartbeater.cc:344] Connected to a master server at 127.6.2.126:45395
I20260318 09:00:14.534060 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:14.534225 9781 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:14.535000 9781 heartbeater.cc:507] Master 127.6.2.126:45395 requested a full tablet report, sending...
I20260318 09:00:14.536895 6153 webserver.cc:492] Webserver started at http://127.6.2.66:36657/ using document root <none> and password file <none>
I20260318 09:00:14.537358 9654 ts_manager.cc:194] Registered new tserver with Master: 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65:35523)
I20260318 09:00:14.537547 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:14.537828 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:14.538143 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:14.539630 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-1-root/instance:
uuid: "4662974da0524731904081e7495830f5"
format_stamp: "Formatted at 2026-03-18 09:00:14 on dist-test-slave-mpvv"
I20260318 09:00:14.539908 9654 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52664
I20260318 09:00:14.544452 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260318 09:00:14.547917 9793 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:14.548713 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260318 09:00:14.549041 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-1-root
uuid: "4662974da0524731904081e7495830f5"
format_stamp: "Formatted at 2026-03-18 09:00:14 on dist-test-slave-mpvv"
I20260318 09:00:14.549319 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:14.587944 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:14.589067 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:14.590534 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 09:00:14.592864 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:14.593096 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:14.593362 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:14.593549 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:14.632874 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.66:46733
I20260318 09:00:14.632998 9855 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.66:46733 every 8 connection(s)
I20260318 09:00:14.637781 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:14.645922 9859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:14.647043 9860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:14.650035 9856 heartbeater.cc:344] Connected to a master server at 127.6.2.126:45395
I20260318 09:00:14.650583 9856 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:14.651484 6153 server_base.cc:1060] running on GCE node
W20260318 09:00:14.651441 9862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:14.651633 9856 heartbeater.cc:507] Master 127.6.2.126:45395 requested a full tablet report, sending...
I20260318 09:00:14.652650 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:14.652915 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:14.653151 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824414653131 us; error 0 us; skew 500 ppm
I20260318 09:00:14.653918 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:14.654066 9654 ts_manager.cc:194] Registered new tserver with Master: 4662974da0524731904081e7495830f5 (127.6.2.66:46733)
I20260318 09:00:14.655970 9654 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52668
I20260318 09:00:14.656832 6153 webserver.cc:492] Webserver started at http://127.6.2.67:38557/ using document root <none> and password file <none>
I20260318 09:00:14.657444 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:14.657696 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:14.658025 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:14.659232 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-2-root/instance:
uuid: "d6cd460fc07a4589b8ccf70c7a88e74e"
format_stamp: "Formatted at 2026-03-18 09:00:14 on dist-test-slave-mpvv"
I20260318 09:00:14.663651 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260318 09:00:14.667001 9867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:14.667996 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260318 09:00:14.668289 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-2-root
uuid: "d6cd460fc07a4589b8ccf70c7a88e74e"
format_stamp: "Formatted at 2026-03-18 09:00:14 on dist-test-slave-mpvv"
I20260318 09:00:14.668674 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:14.690065 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:14.691248 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:14.692739 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 09:00:14.694888 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:14.695122 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:14.695428 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:14.695621 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:14.733728 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.67:45981
I20260318 09:00:14.733826 9929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.67:45981 every 8 connection(s)
I20260318 09:00:14.747380 9930 heartbeater.cc:344] Connected to a master server at 127.6.2.126:45395
I20260318 09:00:14.747861 9930 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:14.748639 9930 heartbeater.cc:507] Master 127.6.2.126:45395 requested a full tablet report, sending...
I20260318 09:00:14.750707 9654 ts_manager.cc:194] Registered new tserver with Master: d6cd460fc07a4589b8ccf70c7a88e74e (127.6.2.67:45981)
I20260318 09:00:14.750936 6153 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014319076s
I20260318 09:00:14.752386 9654 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52684
I20260318 09:00:14.796181 9654 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52700:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260318 09:00:14.836505 9895 tablet_service.cc:1511] Processing CreateTablet for tablet e3ad60417b8b45d0bdd98b7c17b61cf3 (DEFAULT_TABLE table=test-workload [id=eca2353b40df4261a6bb2e3d6cb94d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260318 09:00:14.837872 9895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3ad60417b8b45d0bdd98b7c17b61cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:14.842212 9821 tablet_service.cc:1511] Processing CreateTablet for tablet e3ad60417b8b45d0bdd98b7c17b61cf3 (DEFAULT_TABLE table=test-workload [id=eca2353b40df4261a6bb2e3d6cb94d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260318 09:00:14.843037 9746 tablet_service.cc:1511] Processing CreateTablet for tablet e3ad60417b8b45d0bdd98b7c17b61cf3 (DEFAULT_TABLE table=test-workload [id=eca2353b40df4261a6bb2e3d6cb94d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260318 09:00:14.843664 9821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3ad60417b8b45d0bdd98b7c17b61cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:14.844337 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3ad60417b8b45d0bdd98b7c17b61cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:14.854410 9950 tablet_bootstrap.cc:492] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e: Bootstrap starting.
I20260318 09:00:14.860961 9950 tablet_bootstrap.cc:654] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:14.861806 9952 tablet_bootstrap.cc:492] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Bootstrap starting.
I20260318 09:00:14.864859 9951 tablet_bootstrap.cc:492] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5: Bootstrap starting.
I20260318 09:00:14.868956 9952 tablet_bootstrap.cc:654] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:14.869102 9950 tablet_bootstrap.cc:492] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e: No bootstrap required, opened a new log
I20260318 09:00:14.869709 9950 ts_tablet_manager.cc:1403] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.013s
I20260318 09:00:14.871713 9951 tablet_bootstrap.cc:654] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:14.872875 9950 raft_consensus.cc:359] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:14.873701 9950 raft_consensus.cc:385] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:14.874028 9950 raft_consensus.cc:740] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cd460fc07a4589b8ccf70c7a88e74e, State: Initialized, Role: FOLLOWER
I20260318 09:00:14.874595 9952 tablet_bootstrap.cc:492] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: No bootstrap required, opened a new log
I20260318 09:00:14.875097 9952 ts_tablet_manager.cc:1403] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260318 09:00:14.874925 9950 consensus_queue.cc:260] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:14.876953 9951 tablet_bootstrap.cc:492] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5: No bootstrap required, opened a new log
I20260318 09:00:14.877460 9951 ts_tablet_manager.cc:1403] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260318 09:00:14.877394 9930 heartbeater.cc:499] Master 127.6.2.126:45395 was elected leader, sending a full tablet report...
I20260318 09:00:14.878088 9952 raft_consensus.cc:359] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:14.878831 9952 raft_consensus.cc:385] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:14.879231 9952 raft_consensus.cc:740] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef45de85edb48fd8a35599b4f764a35, State: Initialized, Role: FOLLOWER
I20260318 09:00:14.880125 9952 consensus_queue.cc:260] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:14.880268 9951 raft_consensus.cc:359] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:14.880960 9951 raft_consensus.cc:385] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:14.881258 9951 raft_consensus.cc:740] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4662974da0524731904081e7495830f5, State: Initialized, Role: FOLLOWER
I20260318 09:00:14.882613 9951 consensus_queue.cc:260] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:14.889444 9950 ts_tablet_manager.cc:1434] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260318 09:00:14.889678 9856 heartbeater.cc:499] Master 127.6.2.126:45395 was elected leader, sending a full tablet report...
I20260318 09:00:14.890384 9951 ts_tablet_manager.cc:1434] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260318 09:00:14.893289 9952 ts_tablet_manager.cc:1434] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Time spent starting tablet: real 0.018s user 0.007s sys 0.008s
I20260318 09:00:14.893797 9781 heartbeater.cc:499] Master 127.6.2.126:45395 was elected leader, sending a full tablet report...
I20260318 09:00:14.960140 9956 raft_consensus.cc:493] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 09:00:14.960700 9956 raft_consensus.cc:515] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:14.962934 9956 leader_election.cc:290] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4662974da0524731904081e7495830f5 (127.6.2.66:46733), 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65:35523)
I20260318 09:00:14.971817 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3ad60417b8b45d0bdd98b7c17b61cf3" candidate_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4662974da0524731904081e7495830f5" is_pre_election: true
I20260318 09:00:14.972647 9831 raft_consensus.cc:2468] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6cd460fc07a4589b8ccf70c7a88e74e in term 0.
I20260318 09:00:14.973899 9871 leader_election.cc:304] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4662974da0524731904081e7495830f5, d6cd460fc07a4589b8ccf70c7a88e74e; no voters:
I20260318 09:00:14.974701 9956 raft_consensus.cc:2804] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260318 09:00:14.975035 9956 raft_consensus.cc:493] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 09:00:14.974933 9756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3ad60417b8b45d0bdd98b7c17b61cf3" candidate_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef45de85edb48fd8a35599b4f764a35" is_pre_election: true
I20260318 09:00:14.975585 9956 raft_consensus.cc:3060] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:14.975919 9756 raft_consensus.cc:2468] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6cd460fc07a4589b8ccf70c7a88e74e in term 0.
I20260318 09:00:14.980844 9956 raft_consensus.cc:515] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:14.982527 9956 leader_election.cc:290] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [CANDIDATE]: Term 1 election: Requested vote from peers 4662974da0524731904081e7495830f5 (127.6.2.66:46733), 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65:35523)
I20260318 09:00:14.983510 9831 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3ad60417b8b45d0bdd98b7c17b61cf3" candidate_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4662974da0524731904081e7495830f5"
I20260318 09:00:14.983798 9756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3ad60417b8b45d0bdd98b7c17b61cf3" candidate_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef45de85edb48fd8a35599b4f764a35"
I20260318 09:00:14.984244 9831 raft_consensus.cc:3060] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:14.984499 9756 raft_consensus.cc:3060] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:14.989557 9831 raft_consensus.cc:2468] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6cd460fc07a4589b8ccf70c7a88e74e in term 1.
I20260318 09:00:14.989964 9756 raft_consensus.cc:2468] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6cd460fc07a4589b8ccf70c7a88e74e in term 1.
I20260318 09:00:14.990680 9871 leader_election.cc:304] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4662974da0524731904081e7495830f5, d6cd460fc07a4589b8ccf70c7a88e74e; no voters:
I20260318 09:00:14.991468 9956 raft_consensus.cc:2804] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:14.992451 9956 raft_consensus.cc:697] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 1 LEADER]: Becoming Leader. State: Replica: d6cd460fc07a4589b8ccf70c7a88e74e, State: Running, Role: LEADER
I20260318 09:00:14.993146 9956 consensus_queue.cc:237] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:15.000258 9652 catalog_manager.cc:5671] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e reported cstate change: term changed from 0 to 1, leader changed from <none> to d6cd460fc07a4589b8ccf70c7a88e74e (127.6.2.67). New cstate: current_term: 1 leader_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } } }
I20260318 09:00:15.270747 9821 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:15.271435 9746 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:15.276506 9895 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260318 09:00:15.411574 9956 consensus_queue.cc:1048] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260318 09:00:15.451527 9956 consensus_queue.cc:1048] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
1eb46555646e4a9b80c01034f074471c | 127.6.2.126:45395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
4662974da0524731904081e7495830f5 | 127.6.2.66:46733 | HEALTHY | <none> | 0 | 0
9ef45de85edb48fd8a35599b4f764a35 | 127.6.2.65:35523 | HEALTHY | <none> | 0 | 0
d6cd460fc07a4589b8ccf70c7a88e74e | 127.6.2.67:45981 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet e3ad60417b8b45d0bdd98b7c17b61cf3 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
4662974da0524731904081e7495830f5 (127.6.2.66:46733): RUNNING
d6cd460fc07a4589b8ccf70c7a88e74e (127.6.2.67:45981): RUNNING [LEADER]
9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65:35523): RUNNING
All reported replicas are:
A = 4662974da0524731904081e7495830f5
B = d6cd460fc07a4589b8ccf70c7a88e74e
C = 9ef45de85edb48fd8a35599b4f764a35
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260318 09:00:15.520475 6153 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260318 09:00:15.934381 9821 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:15.940316 9746 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:15.947348 9895 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
1eb46555646e4a9b80c01034f074471c | 127.6.2.126:45395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
4662974da0524731904081e7495830f5 | 127.6.2.66:46733 | HEALTHY | <none> | 0 | 0
9ef45de85edb48fd8a35599b4f764a35 | 127.6.2.65:35523 | HEALTHY | <none> | 0 | 0
d6cd460fc07a4589b8ccf70c7a88e74e | 127.6.2.67:45981 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260318 09:00:16.169262 6153 log_verifier.cc:126] Checking tablet e3ad60417b8b45d0bdd98b7c17b61cf3
I20260318 09:00:16.231452 6153 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e3ad60417b8b45d0bdd98b7c17b61cf3
I20260318 09:00:16.232057 6153 tablet_replica.cc:333] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: stopping tablet replica
I20260318 09:00:16.232621 6153 raft_consensus.cc:2243] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:16.233108 6153 raft_consensus.cc:2272] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:16.453940 9821 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:16.456861 9746 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:16.458925 9895 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
W20260318 09:00:16.596407 9871 consensus_peers.cc:597] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e -> Peer 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65:35523): Couldn't send request to peer 9ef45de85edb48fd8a35599b4f764a35. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260318 09:00:16.597996 10051 raft_consensus.cc:1012] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e: Attempting to remove follower 9ef45de85edb48fd8a35599b4f764a35 from the Raft config. Reason: The tablet replica hosted on peer 9ef45de85edb48fd8a35599b4f764a35 has failed
I20260318 09:00:16.600201 10051 consensus_queue.cc:237] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } }
I20260318 09:00:16.606050 9831 raft_consensus.cc:1275] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 1 FOLLOWER]: Refusing update from remote peer d6cd460fc07a4589b8ccf70c7a88e74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260318 09:00:16.607740 10051 consensus_queue.cc:1048] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260318 09:00:16.616529 10052 raft_consensus.cc:2955] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } }
I20260318 09:00:16.619553 9831 raft_consensus.cc:2955] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } }
I20260318 09:00:16.631618 9654 catalog_manager.cc:5671] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e reported cstate change: config changed from index -1 to 2, VOTER 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65) evicted. New cstate: current_term: 1 leader_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } }
I20260318 09:00:16.664799 9746 tablet_service.cc:1558] Processing DeleteTablet for tablet e3ad60417b8b45d0bdd98b7c17b61cf3 with delete_type TABLET_DATA_TOMBSTONED (TS 9ef45de85edb48fd8a35599b4f764a35 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:36758
I20260318 09:00:16.667248 10060 ts_tablet_manager.cc:1916] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260318 09:00:16.667528 9905 consensus_queue.cc:237] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:16.673583 9831 raft_consensus.cc:1275] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 1 FOLLOWER]: Refusing update from remote peer d6cd460fc07a4589b8ccf70c7a88e74e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260318 09:00:16.673925 9871 consensus_peers.cc:597] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e -> Peer 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65:35523): Couldn't send request to peer 9ef45de85edb48fd8a35599b4f764a35. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260318 09:00:16.675272 10051 consensus_queue.cc:1048] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 09:00:16.682247 10052 raft_consensus.cc:2955] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } } }
I20260318 09:00:16.683463 10060 ts_tablet_manager.cc:1929] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260318 09:00:16.683892 10060 log.cc:1199] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Deleting WAL directory at /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1773824355427976-6153-0/minicluster-data/ts-0-root/wals/e3ad60417b8b45d0bdd98b7c17b61cf3
I20260318 09:00:16.684026 9831 raft_consensus.cc:2955] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } } }
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
1eb46555646e4a9b80c01034f074471c | 127.6.2.126:45395 | HEALTHY
Unusual flags for Master:
I20260318 09:00:16.685868 9641 catalog_manager.cc:5002] TS 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65:35523): tablet e3ad60417b8b45d0bdd98b7c17b61cf3 (table test-workload [id=eca2353b40df4261a6bb2e3d6cb94d79]) successfully deleted
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
4662974da0524731904081e7495830f5 | 127.6.2.66:46733 | HEALTHY | <none> | 0 | 0
9ef45de85edb48fd8a35599b4f764a35 | 127.6.2.65:35523 | HEALTHY | <none> | 0 | 0
d6cd460fc07a4589b8ccf70c7a88e74e | 127.6.2.67:45981 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet e3ad60417b8b45d0bdd98b7c17b61cf3 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
4662974da0524731904081e7495830f5 (127.6.2.66:46733): RUNNING
d6cd460fc07a4589b8ccf70c7a88e74e (127.6.2.67:45981): RUNNING [LEADER]
9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65:35523): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = 4662974da0524731904081e7495830f5
B = d6cd460fc07a4589b8ccf70c7a88e74e
C = 9ef45de85edb48fd8a35599b4f764a35
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
I20260318 09:00:16.692041 9641 catalog_manager.cc:5184] ChangeConfig:ADD_PEER:VOTER RPC for tablet e3ad60417b8b45d0bdd98b7c17b61cf3 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260318 09:00:16.695488 9654 catalog_manager.cc:5671] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e reported cstate change: config changed from index 2 to 3, VOTER 9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65) added. New cstate: current_term: 1 leader_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } } }
I20260318 09:00:16.702828 6153 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260318 09:00:17.068379 9821 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:17.070824 9746 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:17.077987 9895 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260318 09:00:17.255697 10087 ts_tablet_manager.cc:933] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Initiating tablet copy from peer d6cd460fc07a4589b8ccf70c7a88e74e (127.6.2.67:45981)
I20260318 09:00:17.257699 10087 tablet_copy_client.cc:323] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: tablet copy: Beginning tablet copy session from remote peer at address 127.6.2.67:45981
I20260318 09:00:17.268151 9915 tablet_copy_service.cc:140] P d6cd460fc07a4589b8ccf70c7a88e74e: Received BeginTabletCopySession request for tablet e3ad60417b8b45d0bdd98b7c17b61cf3 from peer 9ef45de85edb48fd8a35599b4f764a35 ({username='slave'} at 127.0.0.1:33778)
I20260318 09:00:17.268700 9915 tablet_copy_service.cc:161] P d6cd460fc07a4589b8ccf70c7a88e74e: Beginning new tablet copy session on tablet e3ad60417b8b45d0bdd98b7c17b61cf3 from peer 9ef45de85edb48fd8a35599b4f764a35 at {username='slave'} at 127.0.0.1:33778: session id = 9ef45de85edb48fd8a35599b4f764a35-e3ad60417b8b45d0bdd98b7c17b61cf3
I20260318 09:00:17.274739 9915 tablet_copy_source_session.cc:215] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e: Tablet Copy: opened 0 blocks and 1 log segments
I20260318 09:00:17.276926 10087 ts_tablet_manager.cc:1916] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Deleting tablet data with delete state TABLET_DATA_COPYING
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
1eb46555646e4a9b80c01034f074471c | 127.6.2.126:45395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
4662974da0524731904081e7495830f5 | 127.6.2.66:46733 | HEALTHY | <none> | 0 | 0
9ef45de85edb48fd8a35599b4f764a35 | 127.6.2.65:35523 | HEALTHY | <none> | 0 | 0
d6cd460fc07a4589b8ccf70c7a88e74e | 127.6.2.67:45981 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet e3ad60417b8b45d0bdd98b7c17b61cf3 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
4662974da0524731904081e7495830f5 (127.6.2.66:46733): RUNNING
d6cd460fc07a4589b8ccf70c7a88e74e (127.6.2.67:45981): RUNNING [LEADER]
9ef45de85edb48fd8a35599b4f764a35 (127.6.2.65:35523): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 4662974da0524731904081e7495830f5
B = d6cd460fc07a4589b8ccf70c7a88e74e
C = 9ef45de85edb48fd8a35599b4f764a35
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 3 | Yes
B | A B* C | 1 | 3 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260318 09:00:17.287276 6153 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260318 09:00:17.295692 10087 ts_tablet_manager.cc:1929] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260318 09:00:17.296366 10087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3ad60417b8b45d0bdd98b7c17b61cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:17.301568 10087 tablet_copy_client.cc:806] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: tablet copy: Starting download of 0 data blocks...
I20260318 09:00:17.302026 10087 tablet_copy_client.cc:670] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: tablet copy: Starting download of 1 WAL segments...
I20260318 09:00:17.305310 10087 tablet_copy_client.cc:538] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260318 09:00:17.311957 10087 tablet_bootstrap.cc:492] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Bootstrap starting.
I20260318 09:00:17.328331 10087 tablet_bootstrap.cc:492] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260318 09:00:17.329209 10087 tablet_bootstrap.cc:492] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Bootstrap complete.
I20260318 09:00:17.329804 10087 ts_tablet_manager.cc:1403] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20260318 09:00:17.331972 10087 raft_consensus.cc:359] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:17.332535 10087 raft_consensus.cc:740] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef45de85edb48fd8a35599b4f764a35, State: Initialized, Role: FOLLOWER
I20260318 09:00:17.333169 10087 consensus_queue.cc:260] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4662974da0524731904081e7495830f5" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 46733 } } peers { permanent_uuid: "d6cd460fc07a4589b8ccf70c7a88e74e" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45981 } } peers { permanent_uuid: "9ef45de85edb48fd8a35599b4f764a35" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 35523 } }
I20260318 09:00:17.335318 10087 ts_tablet_manager.cc:1434] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260318 09:00:17.336946 9915 tablet_copy_service.cc:342] P d6cd460fc07a4589b8ccf70c7a88e74e: Request end of tablet copy session 9ef45de85edb48fd8a35599b4f764a35-e3ad60417b8b45d0bdd98b7c17b61cf3 received from {username='slave'} at 127.0.0.1:33778
I20260318 09:00:17.337405 9915 tablet_copy_service.cc:434] P d6cd460fc07a4589b8ccf70c7a88e74e: ending tablet copy session 9ef45de85edb48fd8a35599b4f764a35-e3ad60417b8b45d0bdd98b7c17b61cf3 on tablet e3ad60417b8b45d0bdd98b7c17b61cf3 with peer 9ef45de85edb48fd8a35599b4f764a35
I20260318 09:00:17.645970 9755 raft_consensus.cc:1217] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260318 09:00:17.736023 9821 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:17.737303 9746 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:17.741761 9895 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
1eb46555646e4a9b80c01034f074471c | 127.6.2.126:45395 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
4662974da0524731904081e7495830f5 | 127.6.2.66:46733 | HEALTHY | <none> | 0 | 0
9ef45de85edb48fd8a35599b4f764a35 | 127.6.2.65:35523 | HEALTHY | <none> | 0 | 0
d6cd460fc07a4589b8ccf70c7a88e74e | 127.6.2.67:45981 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260318 09:00:17.943897 6153 log_verifier.cc:126] Checking tablet e3ad60417b8b45d0bdd98b7c17b61cf3
I20260318 09:00:18.008793 6153 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e3ad60417b8b45d0bdd98b7c17b61cf3
I20260318 09:00:18.012197 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 09:00:18.038497 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:18.039315 6153 tablet_replica.cc:333] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35: stopping tablet replica
I20260318 09:00:18.039870 6153 raft_consensus.cc:2243] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:18.040274 6153 raft_consensus.cc:2272] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 9ef45de85edb48fd8a35599b4f764a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:18.061771 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 09:00:18.073094 6153 tablet_server.cc:178] TabletServer@127.6.2.66:0 shutting down...
I20260318 09:00:18.095888 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:18.096741 6153 tablet_replica.cc:333] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5: stopping tablet replica
I20260318 09:00:18.097357 6153 raft_consensus.cc:2243] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:18.097862 6153 raft_consensus.cc:2272] T e3ad60417b8b45d0bdd98b7c17b61cf3 P 4662974da0524731904081e7495830f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:18.118402 6153 tablet_server.cc:195] TabletServer@127.6.2.66:0 shutdown complete.
I20260318 09:00:18.129031 6153 tablet_server.cc:178] TabletServer@127.6.2.67:0 shutting down...
I20260318 09:00:18.152714 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:18.153795 6153 tablet_replica.cc:333] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e: stopping tablet replica
I20260318 09:00:18.154488 6153 raft_consensus.cc:2243] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:18.155366 6153 raft_consensus.cc:2272] T e3ad60417b8b45d0bdd98b7c17b61cf3 P d6cd460fc07a4589b8ccf70c7a88e74e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260318 09:00:18.156780 9871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.2.66:46733: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260318 09:00:18.176657 6153 tablet_server.cc:195] TabletServer@127.6.2.67:0 shutdown complete.
I20260318 09:00:18.187518 6153 master.cc:561] Master@127.6.2.126:45395 shutting down...
I20260318 09:00:18.207118 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:18.207734 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:18.208073 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P 1eb46555646e4a9b80c01034f074471c: stopping tablet replica
I20260318 09:00:18.225934 6153 master.cc:583] Master@127.6.2.126:45395 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (4024 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20260318 09:00:18.258059 6153 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.2.126:40997
I20260318 09:00:18.259465 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:18.266335 10125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:18.266558 10126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:18.268640 10128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:18.269219 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:18.270237 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:18.270408 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:18.270644 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824418270625 us; error 0 us; skew 500 ppm
I20260318 09:00:18.271319 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:18.274381 6153 webserver.cc:492] Webserver started at http://127.6.2.126:37561/ using document root <none> and password file <none>
I20260318 09:00:18.274874 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:18.275060 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:18.275475 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:18.277266 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/master-0-root/instance:
uuid: "cf8b5a38acaf404ab1c7db5956f42f01"
format_stamp: "Formatted at 2026-03-18 09:00:18 on dist-test-slave-mpvv"
I20260318 09:00:18.284150 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260318 09:00:18.288239 10133 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.289199 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260318 09:00:18.289516 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/master-0-root
uuid: "cf8b5a38acaf404ab1c7db5956f42f01"
format_stamp: "Formatted at 2026-03-18 09:00:18 on dist-test-slave-mpvv"
I20260318 09:00:18.289865 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:18.325187 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:18.326452 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:18.365197 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.126:40997
I20260318 09:00:18.365301 10184 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.126:40997 every 8 connection(s)
I20260318 09:00:18.369016 10185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:18.379652 10185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01: Bootstrap starting.
I20260318 09:00:18.384335 10185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:18.388796 10185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01: No bootstrap required, opened a new log
I20260318 09:00:18.390883 10185 raft_consensus.cc:359] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8b5a38acaf404ab1c7db5956f42f01" member_type: VOTER }
I20260318 09:00:18.391485 10185 raft_consensus.cc:385] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:18.391764 10185 raft_consensus.cc:740] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8b5a38acaf404ab1c7db5956f42f01, State: Initialized, Role: FOLLOWER
I20260318 09:00:18.392267 10185 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8b5a38acaf404ab1c7db5956f42f01" member_type: VOTER }
I20260318 09:00:18.392740 10185 raft_consensus.cc:399] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260318 09:00:18.392938 10185 raft_consensus.cc:493] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260318 09:00:18.393141 10185 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:18.398036 10185 raft_consensus.cc:515] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8b5a38acaf404ab1c7db5956f42f01" member_type: VOTER }
I20260318 09:00:18.398515 10185 leader_election.cc:304] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf8b5a38acaf404ab1c7db5956f42f01; no voters:
I20260318 09:00:18.399714 10185 leader_election.cc:290] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260318 09:00:18.400149 10188 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:18.401453 10188 raft_consensus.cc:697] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 1 LEADER]: Becoming Leader. State: Replica: cf8b5a38acaf404ab1c7db5956f42f01, State: Running, Role: LEADER
I20260318 09:00:18.402143 10188 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8b5a38acaf404ab1c7db5956f42f01" member_type: VOTER }
I20260318 09:00:18.402750 10185 sys_catalog.cc:565] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260318 09:00:18.405095 10190 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf8b5a38acaf404ab1c7db5956f42f01. Latest consensus state: current_term: 1 leader_uuid: "cf8b5a38acaf404ab1c7db5956f42f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8b5a38acaf404ab1c7db5956f42f01" member_type: VOTER } }
I20260318 09:00:18.405095 10189 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf8b5a38acaf404ab1c7db5956f42f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8b5a38acaf404ab1c7db5956f42f01" member_type: VOTER } }
I20260318 09:00:18.405817 10190 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:18.405990 10189 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [sys.catalog]: This master's current role is: LEADER
I20260318 09:00:18.410236 10193 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260318 09:00:18.415014 10193 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260318 09:00:18.418996 6153 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260318 09:00:18.424140 10193 catalog_manager.cc:1357] Generated new cluster ID: 9251de259e0a4032a794e898530e1ee3
I20260318 09:00:18.424450 10193 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260318 09:00:18.452620 10193 catalog_manager.cc:1380] Generated new certificate authority record
I20260318 09:00:18.453905 10193 catalog_manager.cc:1514] Loading token signing keys...
I20260318 09:00:18.467795 10193 catalog_manager.cc:6044] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01: Generated new TSK 0
I20260318 09:00:18.468408 10193 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260318 09:00:18.485517 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:18.492053 10206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:18.492333 10207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:18.494392 10209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:18.495056 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:18.495955 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:18.496173 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:18.496327 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824418496313 us; error 0 us; skew 500 ppm
I20260318 09:00:18.496881 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:18.499307 6153 webserver.cc:492] Webserver started at http://127.6.2.65:34113/ using document root <none> and password file <none>
I20260318 09:00:18.499771 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:18.499974 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:18.500244 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:18.501283 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-0-root/instance:
uuid: "b232bf4fe8814b10a195ef242004c93a"
format_stamp: "Formatted at 2026-03-18 09:00:18 on dist-test-slave-mpvv"
I20260318 09:00:18.505612 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260318 09:00:18.508903 10214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.509687 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260318 09:00:18.510001 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-0-root
uuid: "b232bf4fe8814b10a195ef242004c93a"
format_stamp: "Formatted at 2026-03-18 09:00:18 on dist-test-slave-mpvv"
I20260318 09:00:18.510295 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:18.526517 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:18.527822 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:18.529204 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 09:00:18.531597 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:18.531821 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.532078 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:18.532249 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.572727 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.65:39455
I20260318 09:00:18.572865 10276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.65:39455 every 8 connection(s)
I20260318 09:00:18.577756 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:18.585351 10281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:18.586552 10282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:18.589414 6153 server_base.cc:1060] running on GCE node
W20260318 09:00:18.591068 10284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:18.591782 10277 heartbeater.cc:344] Connected to a master server at 127.6.2.126:40997
I20260318 09:00:18.592170 10277 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:18.592252 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:18.592543 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:18.592720 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824418592699 us; error 0 us; skew 500 ppm
I20260318 09:00:18.592974 10277 heartbeater.cc:507] Master 127.6.2.126:40997 requested a full tablet report, sending...
I20260318 09:00:18.593317 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:18.595314 10150 ts_manager.cc:194] Registered new tserver with Master: b232bf4fe8814b10a195ef242004c93a (127.6.2.65:39455)
I20260318 09:00:18.596306 6153 webserver.cc:492] Webserver started at http://127.6.2.66:38477/ using document root <none> and password file <none>
I20260318 09:00:18.597023 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:18.597254 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:18.597540 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:18.597764 10150 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37638
I20260318 09:00:18.598971 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-1-root/instance:
uuid: "4e8e03619071483fb0ceb860f15aa53c"
format_stamp: "Formatted at 2026-03-18 09:00:18 on dist-test-slave-mpvv"
I20260318 09:00:18.603650 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260318 09:00:18.607254 10289 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.608021 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260318 09:00:18.608331 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-1-root
uuid: "4e8e03619071483fb0ceb860f15aa53c"
format_stamp: "Formatted at 2026-03-18 09:00:18 on dist-test-slave-mpvv"
I20260318 09:00:18.608630 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:18.622819 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:18.624073 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:18.625636 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 09:00:18.627912 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:18.628090 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.628378 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:18.628521 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.668668 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.66:38989
I20260318 09:00:18.668787 10351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.66:38989 every 8 connection(s)
I20260318 09:00:18.673436 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:18.681689 10355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:18.682919 10356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:18.686180 10352 heartbeater.cc:344] Connected to a master server at 127.6.2.126:40997
I20260318 09:00:18.686645 10352 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:18.686841 6153 server_base.cc:1060] running on GCE node
W20260318 09:00:18.686897 10358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:18.687776 10352 heartbeater.cc:507] Master 127.6.2.126:40997 requested a full tablet report, sending...
I20260318 09:00:18.688133 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:18.688380 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:18.688591 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824418688571 us; error 0 us; skew 500 ppm
I20260318 09:00:18.689249 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:18.690012 10150 ts_manager.cc:194] Registered new tserver with Master: 4e8e03619071483fb0ceb860f15aa53c (127.6.2.66:38989)
I20260318 09:00:18.692113 10150 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37644
I20260318 09:00:18.692206 6153 webserver.cc:492] Webserver started at http://127.6.2.67:45927/ using document root <none> and password file <none>
I20260318 09:00:18.692909 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:18.693166 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:18.693457 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:18.694558 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-2-root/instance:
uuid: "182596a99485488693b94bd80f4fa642"
format_stamp: "Formatted at 2026-03-18 09:00:18 on dist-test-slave-mpvv"
I20260318 09:00:18.699204 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260318 09:00:18.702659 10363 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.703581 6153 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260318 09:00:18.703994 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-2-root
uuid: "182596a99485488693b94bd80f4fa642"
format_stamp: "Formatted at 2026-03-18 09:00:18 on dist-test-slave-mpvv"
I20260318 09:00:18.704312 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:18.717705 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:18.718933 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:18.720388 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 09:00:18.722689 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:18.722941 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.723254 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:18.723444 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.764091 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.67:45477
I20260318 09:00:18.764195 10425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.67:45477 every 8 connection(s)
I20260318 09:00:18.769021 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260318 09:00:18.776723 10429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:18.778304 10430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260318 09:00:18.780113 10432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260318 09:00:18.781514 6153 server_base.cc:1060] running on GCE node
I20260318 09:00:18.782029 10426 heartbeater.cc:344] Connected to a master server at 127.6.2.126:40997
I20260318 09:00:18.782413 10426 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:18.782503 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260318 09:00:18.782822 6153 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260318 09:00:18.783022 6153 hybrid_clock.cc:648] HybridClock initialized: now 1773824418783004 us; error 0 us; skew 500 ppm
I20260318 09:00:18.783278 10426 heartbeater.cc:507] Master 127.6.2.126:40997 requested a full tablet report, sending...
I20260318 09:00:18.783710 6153 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260318 09:00:18.785674 10150 ts_manager.cc:194] Registered new tserver with Master: 182596a99485488693b94bd80f4fa642 (127.6.2.67:45477)
I20260318 09:00:18.786765 6153 webserver.cc:492] Webserver started at http://127.6.2.68:36809/ using document root <none> and password file <none>
I20260318 09:00:18.787470 6153 fs_manager.cc:362] Metadata directory not provided
I20260318 09:00:18.787709 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260318 09:00:18.787678 10150 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37652
I20260318 09:00:18.788239 6153 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260318 09:00:18.789716 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-3-root/instance:
uuid: "9b89ebceb14247bab3921d4aab345e0a"
format_stamp: "Formatted at 2026-03-18 09:00:18 on dist-test-slave-mpvv"
I20260318 09:00:18.794961 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260318 09:00:18.799024 10437 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.800071 6153 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260318 09:00:18.800352 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-3-root
uuid: "9b89ebceb14247bab3921d4aab345e0a"
format_stamp: "Formatted at 2026-03-18 09:00:18 on dist-test-slave-mpvv"
I20260318 09:00:18.800621 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260318 09:00:18.830037 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260318 09:00:18.831311 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260318 09:00:18.832827 6153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260318 09:00:18.835278 6153 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260318 09:00:18.835583 6153 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.835850 6153 ts_tablet_manager.cc:616] Registered 0 tablets
I20260318 09:00:18.836042 6153 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260318 09:00:18.877616 6153 rpc_server.cc:307] RPC server started. Bound to: 127.6.2.68:32947
I20260318 09:00:18.877715 10499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.2.68:32947 every 8 connection(s)
I20260318 09:00:18.891232 10500 heartbeater.cc:344] Connected to a master server at 127.6.2.126:40997
I20260318 09:00:18.891737 10500 heartbeater.cc:461] Registering TS with master...
I20260318 09:00:18.892486 10500 heartbeater.cc:507] Master 127.6.2.126:40997 requested a full tablet report, sending...
I20260318 09:00:18.894439 10150 ts_manager.cc:194] Registered new tserver with Master: 9b89ebceb14247bab3921d4aab345e0a (127.6.2.68:32947)
I20260318 09:00:18.894915 6153 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.013986709s
I20260318 09:00:18.896112 10150 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37658
I20260318 09:00:19.694824 10352 heartbeater.cc:499] Master 127.6.2.126:40997 was elected leader, sending a full tablet report...
I20260318 09:00:19.726670 10277 heartbeater.cc:499] Master 127.6.2.126:40997 was elected leader, sending a full tablet report...
I20260318 09:00:19.744371 10150 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37670:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260318 09:00:19.790894 10426 heartbeater.cc:499] Master 127.6.2.126:40997 was elected leader, sending a full tablet report...
I20260318 09:00:19.896284 10465 tablet_service.cc:1511] Processing CreateTablet for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 (DEFAULT_TABLE table=test-workload [id=b43e65247b0f4ba79644d3caa32f3204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260318 09:00:19.897719 10465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:19.898804 10500 heartbeater.cc:499] Master 127.6.2.126:40997 was elected leader, sending a full tablet report...
I20260318 09:00:19.899626 10317 tablet_service.cc:1511] Processing CreateTablet for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 (DEFAULT_TABLE table=test-workload [id=b43e65247b0f4ba79644d3caa32f3204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260318 09:00:19.900887 10317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:19.904138 10242 tablet_service.cc:1511] Processing CreateTablet for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 (DEFAULT_TABLE table=test-workload [id=b43e65247b0f4ba79644d3caa32f3204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260318 09:00:19.905287 10242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:19.915040 10524 tablet_bootstrap.cc:492] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a: Bootstrap starting.
I20260318 09:00:19.919113 10525 tablet_bootstrap.cc:492] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c: Bootstrap starting.
I20260318 09:00:19.922096 10524 tablet_bootstrap.cc:654] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:19.922636 10526 tablet_bootstrap.cc:492] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a: Bootstrap starting.
I20260318 09:00:19.925300 10525 tablet_bootstrap.cc:654] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:19.929649 10524 tablet_bootstrap.cc:492] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a: No bootstrap required, opened a new log
I20260318 09:00:19.929899 10526 tablet_bootstrap.cc:654] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a: Neither blocks nor log segments found. Creating new log.
I20260318 09:00:19.930135 10524 ts_tablet_manager.cc:1403] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260318 09:00:19.931032 10525 tablet_bootstrap.cc:492] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c: No bootstrap required, opened a new log
I20260318 09:00:19.931592 10525 ts_tablet_manager.cc:1403] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.002s
I20260318 09:00:19.932682 10524 raft_consensus.cc:359] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 } } peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } }
I20260318 09:00:19.933678 10524 raft_consensus.cc:385] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:19.934001 10524 raft_consensus.cc:740] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b89ebceb14247bab3921d4aab345e0a, State: Initialized, Role: FOLLOWER
I20260318 09:00:19.934592 10525 raft_consensus.cc:359] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 } } peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } }
I20260318 09:00:19.934725 10524 consensus_queue.cc:260] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 } } peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } }
I20260318 09:00:19.935460 10525 raft_consensus.cc:385] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:19.935627 10526 tablet_bootstrap.cc:492] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a: No bootstrap required, opened a new log
I20260318 09:00:19.935904 10525 raft_consensus.cc:740] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8e03619071483fb0ceb860f15aa53c, State: Initialized, Role: FOLLOWER
I20260318 09:00:19.936184 10526 ts_tablet_manager.cc:1403] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260318 09:00:19.936794 10525 consensus_queue.cc:260] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 } } peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } }
I20260318 09:00:19.939468 10526 raft_consensus.cc:359] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 } } peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } }
I20260318 09:00:19.940073 10525 ts_tablet_manager.cc:1434] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260318 09:00:19.940317 10526 raft_consensus.cc:385] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260318 09:00:19.940737 10526 raft_consensus.cc:740] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b232bf4fe8814b10a195ef242004c93a, State: Initialized, Role: FOLLOWER
I20260318 09:00:19.942142 10526 consensus_queue.cc:260] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 } } peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } }
I20260318 09:00:19.942619 10524 ts_tablet_manager.cc:1434] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260318 09:00:19.945153 10526 ts_tablet_manager.cc:1434] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20260318 09:00:20.094353 10532 raft_consensus.cc:493] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260318 09:00:20.094908 10532 raft_consensus.cc:515] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 } } peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } }
I20260318 09:00:20.096906 10532 leader_election.cc:290] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b89ebceb14247bab3921d4aab345e0a (127.6.2.68:32947), 4e8e03619071483fb0ceb860f15aa53c (127.6.2.66:38989)
I20260318 09:00:20.107786 10475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba06f1e5bd1e4d4da50e9a29f1f026b0" candidate_uuid: "b232bf4fe8814b10a195ef242004c93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b89ebceb14247bab3921d4aab345e0a" is_pre_election: true
I20260318 09:00:20.108608 10475 raft_consensus.cc:2468] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b232bf4fe8814b10a195ef242004c93a in term 0.
I20260318 09:00:20.109566 10327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba06f1e5bd1e4d4da50e9a29f1f026b0" candidate_uuid: "b232bf4fe8814b10a195ef242004c93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e8e03619071483fb0ceb860f15aa53c" is_pre_election: true
I20260318 09:00:20.109623 10215 leader_election.cc:304] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b89ebceb14247bab3921d4aab345e0a, b232bf4fe8814b10a195ef242004c93a; no voters:
I20260318 09:00:20.110317 10327 raft_consensus.cc:2468] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b232bf4fe8814b10a195ef242004c93a in term 0.
I20260318 09:00:20.110598 10532 raft_consensus.cc:2804] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260318 09:00:20.110988 10532 raft_consensus.cc:493] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260318 09:00:20.111423 10532 raft_consensus.cc:3060] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:20.116355 10532 raft_consensus.cc:515] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 } } peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } }
I20260318 09:00:20.117940 10532 leader_election.cc:290] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [CANDIDATE]: Term 1 election: Requested vote from peers 9b89ebceb14247bab3921d4aab345e0a (127.6.2.68:32947), 4e8e03619071483fb0ceb860f15aa53c (127.6.2.66:38989)
I20260318 09:00:20.118844 10475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba06f1e5bd1e4d4da50e9a29f1f026b0" candidate_uuid: "b232bf4fe8814b10a195ef242004c93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b89ebceb14247bab3921d4aab345e0a"
I20260318 09:00:20.119021 10327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba06f1e5bd1e4d4da50e9a29f1f026b0" candidate_uuid: "b232bf4fe8814b10a195ef242004c93a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e8e03619071483fb0ceb860f15aa53c"
I20260318 09:00:20.119370 10475 raft_consensus.cc:3060] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:20.119642 10327 raft_consensus.cc:3060] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 0 FOLLOWER]: Advancing to term 1
I20260318 09:00:20.124614 10475 raft_consensus.cc:2468] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b232bf4fe8814b10a195ef242004c93a in term 1.
I20260318 09:00:20.124650 10327 raft_consensus.cc:2468] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b232bf4fe8814b10a195ef242004c93a in term 1.
I20260318 09:00:20.125785 10215 leader_election.cc:304] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b89ebceb14247bab3921d4aab345e0a, b232bf4fe8814b10a195ef242004c93a; no voters:
I20260318 09:00:20.126466 10532 raft_consensus.cc:2804] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 1 FOLLOWER]: Leader election won for term 1
I20260318 09:00:20.127555 10532 raft_consensus.cc:697] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 1 LEADER]: Becoming Leader. State: Replica: b232bf4fe8814b10a195ef242004c93a, State: Running, Role: LEADER
I20260318 09:00:20.128398 10532 consensus_queue.cc:237] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 } } peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } }
I20260318 09:00:20.136731 10149 catalog_manager.cc:5671] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a reported cstate change: term changed from 0 to 1, leader changed from <none> to b232bf4fe8814b10a195ef242004c93a (127.6.2.65). New cstate: current_term: 1 leader_uuid: "b232bf4fe8814b10a195ef242004c93a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } health_report { overall_health: HEALTHY } } }
I20260318 09:00:20.389154 10391 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:20.394156 10465 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:20.400686 10242 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260318 09:00:20.406036 10317 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:20.542143 10532 consensus_queue.cc:1048] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260318 09:00:20.595453 10535 consensus_queue.cc:1048] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b89ebceb14247bab3921d4aab345e0a" member_type: VOTER last_known_addr { host: "127.6.2.68" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
cf8b5a38acaf404ab1c7db5956f42f01 | 127.6.2.126:40997 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
182596a99485488693b94bd80f4fa642 | 127.6.2.67:45477 | HEALTHY | <none> | 0 | 0
4e8e03619071483fb0ceb860f15aa53c | 127.6.2.66:38989 | HEALTHY | <none> | 0 | 0
9b89ebceb14247bab3921d4aab345e0a | 127.6.2.68:32947 | HEALTHY | <none> | 0 | 0
b232bf4fe8814b10a195ef242004c93a | 127.6.2.65:39455 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
9b89ebceb14247bab3921d4aab345e0a (127.6.2.68:32947): RUNNING
4e8e03619071483fb0ceb860f15aa53c (127.6.2.66:38989): RUNNING
b232bf4fe8814b10a195ef242004c93a (127.6.2.65:39455): RUNNING [LEADER]
All reported replicas are:
A = 9b89ebceb14247bab3921d4aab345e0a
B = 4e8e03619071483fb0ceb860f15aa53c
C = b232bf4fe8814b10a195ef242004c93a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Small | 182596a99485488693b94bd80f4fa642 | 127.6.2.67:45477 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260318 09:00:20.687557 6153 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
W20260318 09:00:20.762236 10348 debug-util.cc:398] Leaking SignalData structure 0x7b08000ccec0 after lost signal to thread 6156
I20260318 09:00:21.100859 10391 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:21.109416 10317 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:21.122673 10242 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260318 09:00:21.127525 10465 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
cf8b5a38acaf404ab1c7db5956f42f01 | 127.6.2.126:40997 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
182596a99485488693b94bd80f4fa642 | 127.6.2.67:45477 | HEALTHY | <none> | 0 | 0
4e8e03619071483fb0ceb860f15aa53c | 127.6.2.66:38989 | HEALTHY | <none> | 0 | 0
9b89ebceb14247bab3921d4aab345e0a | 127.6.2.68:32947 | HEALTHY | <none> | 0 | 0
b232bf4fe8814b10a195ef242004c93a | 127.6.2.65:39455 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Small | 182596a99485488693b94bd80f4fa642 | 127.6.2.67:45477 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260318 09:00:21.448524 6153 log_verifier.cc:126] Checking tablet ba06f1e5bd1e4d4da50e9a29f1f026b0
I20260318 09:00:21.522675 6153 log_verifier.cc:177] Verified matching terms for 1 ops in tablet ba06f1e5bd1e4d4da50e9a29f1f026b0
I20260318 09:00:21.523270 6153 tablet_replica.cc:333] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a: stopping tablet replica
I20260318 09:00:21.523808 6153 raft_consensus.cc:2243] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:21.524268 6153 raft_consensus.cc:2272] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:21.679021 10215 consensus_queue.cc:579] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: The tablet replica hosted on peer 9b89ebceb14247bab3921d4aab345e0a has failed
W20260318 09:00:21.679584 10215 consensus_peers.cc:597] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a -> Peer 9b89ebceb14247bab3921d4aab345e0a (127.6.2.68:32947): Couldn't send request to peer 9b89ebceb14247bab3921d4aab345e0a. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260318 09:00:21.687960 10252 consensus_queue.cc:237] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } }
I20260318 09:00:21.693732 10327 raft_consensus.cc:1275] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 1 FOLLOWER]: Refusing update from remote peer b232bf4fe8814b10a195ef242004c93a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260318 09:00:21.695345 10607 consensus_queue.cc:1048] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260318 09:00:21.706085 10606 raft_consensus.cc:2955] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9b89ebceb14247bab3921d4aab345e0a (127.6.2.68) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } }
I20260318 09:00:21.708232 10327 raft_consensus.cc:2955] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 9b89ebceb14247bab3921d4aab345e0a (127.6.2.68) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } }
I20260318 09:00:21.717761 10136 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260318 09:00:21.720080 10149 catalog_manager.cc:5671] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a reported cstate change: config changed from index -1 to 2, VOTER 9b89ebceb14247bab3921d4aab345e0a (127.6.2.68) evicted. New cstate: current_term: 1 leader_uuid: "b232bf4fe8814b10a195ef242004c93a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } health_report { overall_health: HEALTHY } } }
I20260318 09:00:21.747167 10465 tablet_service.cc:1558] Processing DeleteTablet for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 with delete_type TABLET_DATA_TOMBSTONED (TS 9b89ebceb14247bab3921d4aab345e0a not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:47916
I20260318 09:00:21.750226 10633 ts_tablet_manager.cc:1916] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260318 09:00:21.754006 10252 consensus_queue.cc:237] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: NON_VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: true } }
I20260318 09:00:21.765343 10327 raft_consensus.cc:1275] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 1 FOLLOWER]: Refusing update from remote peer b232bf4fe8814b10a195ef242004c93a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260318 09:00:21.767206 10535 consensus_queue.cc:1048] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260318 09:00:21.775100 10606 raft_consensus.cc:2955] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 182596a99485488693b94bd80f4fa642 (127.6.2.67) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: NON_VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: true } } }
I20260318 09:00:21.785498 10327 raft_consensus.cc:2955] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 182596a99485488693b94bd80f4fa642 (127.6.2.67) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: NON_VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: true } } }
I20260318 09:00:21.794718 10633 ts_tablet_manager.cc:1929] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260318 09:00:21.800876 10633 log.cc:1199] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 9b89ebceb14247bab3921d4aab345e0a: Deleting WAL directory at /tmp/dist-test-task2XDaNv/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1773824355427976-6153-0/minicluster-data/ts-3-root/wals/ba06f1e5bd1e4d4da50e9a29f1f026b0
I20260318 09:00:21.801798 10136 catalog_manager.cc:5184] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260318 09:00:21.804911 10134 catalog_manager.cc:5002] TS 9b89ebceb14247bab3921d4aab345e0a (127.6.2.68:32947): tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 (table test-workload [id=b43e65247b0f4ba79644d3caa32f3204]) successfully deleted
I20260318 09:00:21.808580 10149 catalog_manager.cc:5671] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a reported cstate change: config changed from index 2 to 3, NON_VOTER 182596a99485488693b94bd80f4fa642 (127.6.2.67) added. New cstate: current_term: 1 leader_uuid: "b232bf4fe8814b10a195ef242004c93a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: NON_VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260318 09:00:21.856911 10218 consensus_peers.cc:597] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a -> Peer 182596a99485488693b94bd80f4fa642 (127.6.2.67:45477): Couldn't send request to peer 182596a99485488693b94bd80f4fa642. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba06f1e5bd1e4d4da50e9a29f1f026b0. This is attempt 1: this message will repeat every 5th retry.
I20260318 09:00:21.859004 10391 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:21.898444 10317 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:21.912189 10465 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:21.916937 10242 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260318 09:00:22.208683 10646 ts_tablet_manager.cc:933] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: Initiating tablet copy from peer b232bf4fe8814b10a195ef242004c93a (127.6.2.65:39455)
I20260318 09:00:22.210692 10646 tablet_copy_client.cc:323] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: tablet copy: Beginning tablet copy session from remote peer at address 127.6.2.65:39455
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
cf8b5a38acaf404ab1c7db5956f42f01 | 127.6.2.126:40997 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
182596a99485488693b94bd80f4fa642 | 127.6.2.67:45477 | HEALTHY | <none> | 0 | 0
4e8e03619071483fb0ceb860f15aa53c | 127.6.2.66:38989 | HEALTHY | <none> | 0 | 0
9b89ebceb14247bab3921d4aab345e0a | 127.6.2.68:32947 | HEALTHY | <none> | 0 | 0
b232bf4fe8814b10a195ef242004c93a | 127.6.2.65:39455 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
4e8e03619071483fb0ceb860f15aa53c (127.6.2.66:38989): RUNNING
b232bf4fe8814b10a195ef242004c93a (127.6.2.65:39455): RUNNING [LEADER]
All reported replicas are:
A = 4e8e03619071483fb0ceb860f15aa53c
B = b232bf4fe8814b10a195ef242004c93a
C = 182596a99485488693b94bd80f4fa642
The consensus matrix is:
I20260318 09:00:22.226866 10262 tablet_copy_service.cc:140] P b232bf4fe8814b10a195ef242004c93a: Received BeginTabletCopySession request for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 from peer 182596a99485488693b94bd80f4fa642 ({username='slave'} at 127.0.0.1:52306)
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* | | | Yes
A | A B* C~ | 1 | 3 | Yes
B | A B* C~ | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
I20260318 09:00:22.227581 10262 tablet_copy_service.cc:161] P b232bf4fe8814b10a195ef242004c93a: Beginning new tablet copy session on tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 from peer 182596a99485488693b94bd80f4fa642 at {username='slave'} at 127.0.0.1:52306: session id = 182596a99485488693b94bd80f4fa642-ba06f1e5bd1e4d4da50e9a29f1f026b0
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 2
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260318 09:00:22.236527 10262 tablet_copy_source_session.cc:215] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a: Tablet Copy: opened 0 blocks and 1 log segments
I20260318 09:00:22.239496 6153 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260318 09:00:22.239732 10646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba06f1e5bd1e4d4da50e9a29f1f026b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260318 09:00:22.251484 10646 tablet_copy_client.cc:806] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: tablet copy: Starting download of 0 data blocks...
I20260318 09:00:22.252045 10646 tablet_copy_client.cc:670] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: tablet copy: Starting download of 1 WAL segments...
I20260318 09:00:22.255728 10646 tablet_copy_client.cc:538] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260318 09:00:22.262216 10646 tablet_bootstrap.cc:492] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: Bootstrap starting.
I20260318 09:00:22.278441 10646 tablet_bootstrap.cc:492] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260318 09:00:22.279286 10646 tablet_bootstrap.cc:492] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: Bootstrap complete.
I20260318 09:00:22.279858 10646 ts_tablet_manager.cc:1403] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.008s
I20260318 09:00:22.281940 10646 raft_consensus.cc:359] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: NON_VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: true } }
I20260318 09:00:22.282524 10646 raft_consensus.cc:740] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 182596a99485488693b94bd80f4fa642, State: Initialized, Role: LEARNER
I20260318 09:00:22.283023 10646 consensus_queue.cc:260] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: NON_VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: true } }
I20260318 09:00:22.286171 10646 ts_tablet_manager.cc:1434] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260318 09:00:22.288079 10262 tablet_copy_service.cc:342] P b232bf4fe8814b10a195ef242004c93a: Request end of tablet copy session 182596a99485488693b94bd80f4fa642-ba06f1e5bd1e4d4da50e9a29f1f026b0 received from {username='slave'} at 127.0.0.1:52306
I20260318 09:00:22.288504 10262 tablet_copy_service.cc:434] P b232bf4fe8814b10a195ef242004c93a: ending tablet copy session 182596a99485488693b94bd80f4fa642-ba06f1e5bd1e4d4da50e9a29f1f026b0 on tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 with peer 182596a99485488693b94bd80f4fa642
I20260318 09:00:22.656389 10317 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:22.658423 10391 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:22.668540 10242 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260318 09:00:22.671082 10465 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:22.796541 10401 raft_consensus.cc:1217] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
cf8b5a38acaf404ab1c7db5956f42f01 | 127.6.2.126:40997 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
182596a99485488693b94bd80f4fa642 | 127.6.2.67:45477 | HEALTHY | <none> | 0 | 0
4e8e03619071483fb0ceb860f15aa53c | 127.6.2.66:38989 | HEALTHY | <none> | 0 | 0
9b89ebceb14247bab3921d4aab345e0a | 127.6.2.68:32947 | HEALTHY | <none> | 0 | 0
b232bf4fe8814b10a195ef242004c93a | 127.6.2.65:39455 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet ba06f1e5bd1e4d4da50e9a29f1f026b0 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
4e8e03619071483fb0ceb860f15aa53c (127.6.2.66:38989): RUNNING
b232bf4fe8814b10a195ef242004c93a (127.6.2.65:39455): RUNNING [LEADER]
182596a99485488693b94bd80f4fa642 (127.6.2.67:45477): RUNNING [NONVOTER]
All reported replicas are:
A = 4e8e03619071483fb0ceb860f15aa53c
B = b232bf4fe8814b10a195ef242004c93a
C = 182596a99485488693b94bd80f4fa642
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 1 | 3 | Yes
B | A B* C~ | 1 | 3 | Yes
C | A B C~ | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Small | 9b89ebceb14247bab3921d4aab345e0a | 127.6.2.68:32947 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260318 09:00:22.991461 6153 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260318 09:00:23.195112 10680 raft_consensus.cc:1064] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a: attempting to promote NON_VOTER 182596a99485488693b94bd80f4fa642 to VOTER
I20260318 09:00:23.196851 10680 consensus_queue.cc:237] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: false } }
I20260318 09:00:23.203560 10401 raft_consensus.cc:1275] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642 [term 1 LEARNER]: Refusing update from remote peer b232bf4fe8814b10a195ef242004c93a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260318 09:00:23.203814 10327 raft_consensus.cc:1275] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 1 FOLLOWER]: Refusing update from remote peer b232bf4fe8814b10a195ef242004c93a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260318 09:00:23.205379 10606 consensus_queue.cc:1048] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260318 09:00:23.207410 10606 consensus_queue.cc:1048] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260318 09:00:23.214959 10683 raft_consensus.cc:2955] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 182596a99485488693b94bd80f4fa642 (127.6.2.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: false } } }
I20260318 09:00:23.216795 10401 raft_consensus.cc:2955] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 182596a99485488693b94bd80f4fa642 (127.6.2.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: false } } }
I20260318 09:00:23.231395 10327 raft_consensus.cc:2955] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 182596a99485488693b94bd80f4fa642 (127.6.2.67) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: false } } }
I20260318 09:00:23.233994 10150 catalog_manager.cc:5671] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642 reported cstate change: config changed from index 3 to 4, 182596a99485488693b94bd80f4fa642 (127.6.2.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b232bf4fe8814b10a195ef242004c93a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e8e03619071483fb0ceb860f15aa53c" member_type: VOTER last_known_addr { host: "127.6.2.66" port: 38989 } } peers { permanent_uuid: "b232bf4fe8814b10a195ef242004c93a" member_type: VOTER last_known_addr { host: "127.6.2.65" port: 39455 } } peers { permanent_uuid: "182596a99485488693b94bd80f4fa642" member_type: VOTER last_known_addr { host: "127.6.2.67" port: 45477 } attrs { promote: false } } }
I20260318 09:00:23.507797 10391 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:23.515750 10317 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:23.520586 10465 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260318 09:00:23.525792 10242 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
cf8b5a38acaf404ab1c7db5956f42f01 | 127.6.2.126:40997 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
182596a99485488693b94bd80f4fa642 | 127.6.2.67:45477 | HEALTHY | <none> | 0 | 0
4e8e03619071483fb0ceb860f15aa53c | 127.6.2.66:38989 | HEALTHY | <none> | 0 | 0
9b89ebceb14247bab3921d4aab345e0a | 127.6.2.68:32947 | HEALTHY | <none> | 0 | 0
b232bf4fe8814b10a195ef242004c93a | 127.6.2.65:39455 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+------------------+---------------
Small | 9b89ebceb14247bab3921d4aab345e0a | 127.6.2.68:32947 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260318 09:00:23.844975 6153 log_verifier.cc:126] Checking tablet ba06f1e5bd1e4d4da50e9a29f1f026b0
I20260318 09:00:23.922715 6153 log_verifier.cc:177] Verified matching terms for 4 ops in tablet ba06f1e5bd1e4d4da50e9a29f1f026b0
I20260318 09:00:23.925040 6153 tablet_server.cc:178] TabletServer@127.6.2.65:0 shutting down...
I20260318 09:00:23.957413 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:23.958087 6153 tablet_replica.cc:333] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a: stopping tablet replica
I20260318 09:00:23.958693 6153 raft_consensus.cc:2243] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:23.959563 6153 raft_consensus.cc:2272] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P b232bf4fe8814b10a195ef242004c93a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:23.972880 6153 tablet_server.cc:195] TabletServer@127.6.2.65:0 shutdown complete.
I20260318 09:00:23.991189 6153 tablet_server.cc:178] TabletServer@127.6.2.66:0 shutting down...
I20260318 09:00:24.025466 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:24.026973 6153 tablet_replica.cc:333] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c: stopping tablet replica
I20260318 09:00:24.027997 6153 raft_consensus.cc:2243] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:24.028700 6153 raft_consensus.cc:2272] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 4e8e03619071483fb0ceb860f15aa53c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:24.410326 6153 tablet_server.cc:195] TabletServer@127.6.2.66:0 shutdown complete.
I20260318 09:00:24.419792 6153 tablet_server.cc:178] TabletServer@127.6.2.67:0 shutting down...
I20260318 09:00:24.440290 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:24.440869 6153 tablet_replica.cc:333] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642: stopping tablet replica
I20260318 09:00:24.441319 6153 raft_consensus.cc:2243] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260318 09:00:24.441730 6153 raft_consensus.cc:2272] T ba06f1e5bd1e4d4da50e9a29f1f026b0 P 182596a99485488693b94bd80f4fa642 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:24.460929 6153 tablet_server.cc:195] TabletServer@127.6.2.67:0 shutdown complete.
I20260318 09:00:24.469916 6153 tablet_server.cc:178] TabletServer@127.6.2.68:0 shutting down...
I20260318 09:00:24.488816 6153 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260318 09:00:24.505942 6153 tablet_server.cc:195] TabletServer@127.6.2.68:0 shutdown complete.
I20260318 09:00:24.515208 6153 master.cc:561] Master@127.6.2.126:40997 shutting down...
I20260318 09:00:24.531524 6153 raft_consensus.cc:2243] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 1 LEADER]: Raft consensus shutting down.
I20260318 09:00:24.532073 6153 raft_consensus.cc:2272] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260318 09:00:24.532383 6153 tablet_replica.cc:333] T 00000000000000000000000000000000 P cf8b5a38acaf404ab1c7db5956f42f01: stopping tablet replica
I20260318 09:00:24.550473 6153 master.cc:583] Master@127.6.2.126:40997 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (6323 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (10348 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (68985 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20260318 09:00:24.588510 6153 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~11 seconds ago
I20260318 09:00:24.588883 6153 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 56 messages since previous log ~61 seconds ago
I20260318 09:00:24.589228 6153 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2379: suppressed but not reported on 8 messages since previous log ~16 seconds ago