[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 18:50:05.378126 29046 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 1.361s user 0.546s sys 0.812s
W20260810 18:50:05.378706 29046 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 1.362s user 0.546s sys 0.812s
I20260810 18:50:05.387454 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:44219
I20260810 18:50:05.389917 29046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:50:05.391428 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:05.422734 29061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:05.427277 29065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:05.425616 29060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:05.517948 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:05.519013 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:05.519225 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:05.519368 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387805519355 us; error 0 us; skew 500 ppm
I20260810 18:50:05.520133 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:05.528439 29046 webserver.cc:460] Webserver started at http://127.28.93.190:38289/ using document root <none> and password file <none>
I20260810 18:50:05.529392 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:05.529592 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:05.529981 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:05.536264 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "a9410b30c55f406ba5159d2e8242b06e"
format_stamp: "Formatted at 2026-08-10 18:50:05 on dist-test-slave-nj21"
I20260810 18:50:05.544747 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s
I20260810 18:50:05.550175 29071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.551252 29046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260810 18:50:05.551586 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "a9410b30c55f406ba5159d2e8242b06e"
format_stamp: "Formatted at 2026-08-10 18:50:05 on dist-test-slave-nj21"
I20260810 18:50:05.551935 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:05.570233 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:05.571908 29046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:50:05.572369 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:05.672369 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:44219
I20260810 18:50:05.672422 29152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:44219 every 8 connection(s)
I20260810 18:50:05.682152 29153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:05.709566 29153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e: Bootstrap starting.
I20260810 18:50:05.717590 29153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:05.720067 29153 log.cc:824] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:50:05.725950 29153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e: No bootstrap required, opened a new log
I20260810 18:50:05.752523 29153 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9410b30c55f406ba5159d2e8242b06e" member_type: VOTER }
I20260810 18:50:05.753188 29153 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:05.753441 29153 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9410b30c55f406ba5159d2e8242b06e, State: Initialized, Role: FOLLOWER
I20260810 18:50:05.754463 29153 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9410b30c55f406ba5159d2e8242b06e" member_type: VOTER }
I20260810 18:50:05.755110 29153 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:05.755399 29153 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:05.755777 29153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:05.759905 29153 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9410b30c55f406ba5159d2e8242b06e" member_type: VOTER }
I20260810 18:50:05.760586 29153 leader_election.cc:302] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9410b30c55f406ba5159d2e8242b06e; no voters:
I20260810 18:50:05.763011 29153 leader_election.cc:288] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:05.763459 29164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:05.766744 29164 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 1 LEADER]: Becoming Leader. State: Replica: a9410b30c55f406ba5159d2e8242b06e, State: Running, Role: LEADER
I20260810 18:50:05.767563 29164 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9410b30c55f406ba5159d2e8242b06e" member_type: VOTER }
I20260810 18:50:05.768379 29153 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:05.778601 29165 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9410b30c55f406ba5159d2e8242b06e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9410b30c55f406ba5159d2e8242b06e" member_type: VOTER } }
I20260810 18:50:05.780041 29165 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:05.779901 29168 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9410b30c55f406ba5159d2e8242b06e. Latest consensus state: current_term: 1 leader_uuid: "a9410b30c55f406ba5159d2e8242b06e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9410b30c55f406ba5159d2e8242b06e" member_type: VOTER } }
I20260810 18:50:05.780575 29168 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:05.784703 29175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:05.798578 29175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:05.804960 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:05.818548 29175 catalog_manager.cc:1347] Generated new cluster ID: c360f34d4093441293fc419e0ca62b28
I20260810 18:50:05.818833 29175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:05.842336 29175 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:05.844444 29175 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:05.864446 29175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e: Generated new TSK 0
I20260810 18:50:05.865471 29175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:05.874890 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:05.883870 29190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:05.886261 29191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:05.891499 29194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:05.891690 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:05.892480 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:05.892663 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:05.892848 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387805892825 us; error 0 us; skew 500 ppm
I20260810 18:50:05.893419 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:05.897805 29046 webserver.cc:460] Webserver started at http://127.28.93.129:45109/ using document root <none> and password file <none>
I20260810 18:50:05.898298 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:05.898480 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:05.898737 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:05.900094 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "3587b2902bc647269da683b5195eed6c"
format_stamp: "Formatted at 2026-08-10 18:50:05 on dist-test-slave-nj21"
I20260810 18:50:05.906112 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 18:50:05.910739 29203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.911574 29046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 18:50:05.911873 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "3587b2902bc647269da683b5195eed6c"
format_stamp: "Formatted at 2026-08-10 18:50:05 on dist-test-slave-nj21"
I20260810 18:50:05.912151 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:05.929203 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:05.930472 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:05.934290 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:05.939426 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:05.939636 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:05.939954 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:05.940128 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:06.073402 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:41077
I20260810 18:50:06.074319 29292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:41077 every 8 connection(s)
I20260810 18:50:06.092507 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:06.110575 29299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:06.111706 29300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:06.116710 29304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:06.119017 29293 heartbeater.cc:347] Connected to a master server at 127.28.93.190:44219
I20260810 18:50:06.119514 29293 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:06.120662 29293 heartbeater.cc:511] Master 127.28.93.190:44219 requested a full tablet report, sending...
I20260810 18:50:06.121788 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:06.122661 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:06.122879 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:06.123054 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387806123038 us; error 0 us; skew 500 ppm
I20260810 18:50:06.123543 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:06.125237 29099 ts_manager.cc:194] Registered new tserver with Master: 3587b2902bc647269da683b5195eed6c (127.28.93.129:41077)
I20260810 18:50:06.126362 29046 webserver.cc:460] Webserver started at http://127.28.93.130:39239/ using document root <none> and password file <none>
I20260810 18:50:06.126816 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:06.126992 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:06.127249 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:06.128305 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/ts-1-root/instance:
uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda"
format_stamp: "Formatted at 2026-08-10 18:50:06 on dist-test-slave-nj21"
I20260810 18:50:06.129091 29099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57828
I20260810 18:50:06.134236 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 18:50:06.139117 29316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:06.139943 29046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:50:06.140187 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/ts-1-root
uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda"
format_stamp: "Formatted at 2026-08-10 18:50:06 on dist-test-slave-nj21"
I20260810 18:50:06.140458 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786387803638833-29046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:06.171458 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:06.172936 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:06.175995 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:06.180091 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:06.180275 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:06.180523 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:06.180660 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:06.234087 29293 heartbeater.cc:503] Master 127.28.93.190:44219 was elected leader, sending a full tablet report...
I20260810 18:50:06.310019 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.130:43759
I20260810 18:50:06.310220 29412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.130:43759 every 8 connection(s)
I20260810 18:50:06.333959 29415 heartbeater.cc:347] Connected to a master server at 127.28.93.190:44219
I20260810 18:50:06.334291 29415 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:06.335139 29415 heartbeater.cc:511] Master 127.28.93.190:44219 requested a full tablet report, sending...
I20260810 18:50:06.337097 29099 ts_manager.cc:194] Registered new tserver with Master: 8c7fc479c1ed4c37a3fc2914b5a6efda (127.28.93.130:43759)
I20260810 18:50:06.337206 29046 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.018949734s
I20260810 18:50:06.339298 29099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57842
I20260810 18:50:06.375708 29099 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:50:06.398380 29099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260810 18:50:06.444933 29415 heartbeater.cc:503] Master 127.28.93.190:44219 was elected leader, sending a full tablet report...
I20260810 18:50:06.469096 29357 tablet_service.cc:1467] Processing CreateTablet for tablet 85cb4e5c60a6430a98c8783d0233e531 (DEFAULT_TABLE table=test-table [id=e533e96ee9e0452eb1abcee0f6c3614a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:06.469648 29237 tablet_service.cc:1467] Processing CreateTablet for tablet 3ebdcb16d19c413d945efe3dd98e527f (DEFAULT_TABLE table=test-table [id=e533e96ee9e0452eb1abcee0f6c3614a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:06.471072 29237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ebdcb16d19c413d945efe3dd98e527f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.471197 29238 tablet_service.cc:1467] Processing CreateTablet for tablet 85cb4e5c60a6430a98c8783d0233e531 (DEFAULT_TABLE table=test-table [id=e533e96ee9e0452eb1abcee0f6c3614a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:06.472057 29357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cb4e5c60a6430a98c8783d0233e531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.472594 29356 tablet_service.cc:1467] Processing CreateTablet for tablet 3ebdcb16d19c413d945efe3dd98e527f (DEFAULT_TABLE table=test-table [id=e533e96ee9e0452eb1abcee0f6c3614a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:06.474177 29238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85cb4e5c60a6430a98c8783d0233e531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.475548 29356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ebdcb16d19c413d945efe3dd98e527f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:06.501226 29437 tablet_bootstrap.cc:492] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda: Bootstrap starting.
I20260810 18:50:06.505297 29438 tablet_bootstrap.cc:492] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c: Bootstrap starting.
I20260810 18:50:06.508297 29437 tablet_bootstrap.cc:654] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.522169 29438 tablet_bootstrap.cc:654] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.528486 29437 tablet_bootstrap.cc:492] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda: No bootstrap required, opened a new log
I20260810 18:50:06.529193 29437 ts_tablet_manager.cc:1397] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.008s
I20260810 18:50:06.534852 29437 raft_consensus.cc:348] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:06.537353 29437 raft_consensus.cc:729] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7fc479c1ed4c37a3fc2914b5a6efda, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.538681 29437 consensus_queue.cc:260] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:06.551489 29437 ts_tablet_manager.cc:1428] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda: Time spent starting tablet: real 0.022s user 0.014s sys 0.001s
I20260810 18:50:06.552887 29437 tablet_bootstrap.cc:492] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda: Bootstrap starting.
I20260810 18:50:06.564450 29438 tablet_bootstrap.cc:492] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c: No bootstrap required, opened a new log
I20260810 18:50:06.568595 29438 ts_tablet_manager.cc:1397] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.021s
W20260810 18:50:06.571223 29416 tablet.cc:2369] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:50:06.577211 29437 tablet_bootstrap.cc:654] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.579442 29438 raft_consensus.cc:348] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:06.583880 29438 raft_consensus.cc:729] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3587b2902bc647269da683b5195eed6c, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.585589 29438 consensus_queue.cc:260] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:06.605480 29438 ts_tablet_manager.cc:1428] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c: Time spent starting tablet: real 0.036s user 0.011s sys 0.013s
I20260810 18:50:06.607033 29437 tablet_bootstrap.cc:492] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda: No bootstrap required, opened a new log
I20260810 18:50:06.607278 29438 tablet_bootstrap.cc:492] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c: Bootstrap starting.
I20260810 18:50:06.607752 29437 ts_tablet_manager.cc:1397] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda: Time spent bootstrapping tablet: real 0.055s user 0.018s sys 0.011s
I20260810 18:50:06.614210 29437 raft_consensus.cc:348] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } } peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } }
I20260810 18:50:06.614897 29437 raft_consensus.cc:729] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7fc479c1ed4c37a3fc2914b5a6efda, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.615901 29437 consensus_queue.cc:260] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } } peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } }
I20260810 18:50:06.619879 29437 ts_tablet_manager.cc:1428] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260810 18:50:06.625051 29438 tablet_bootstrap.cc:654] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:06.634714 29438 tablet_bootstrap.cc:492] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c: No bootstrap required, opened a new log
I20260810 18:50:06.635219 29438 ts_tablet_manager.cc:1397] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.001s
I20260810 18:50:06.640156 29438 raft_consensus.cc:348] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } } peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } }
I20260810 18:50:06.641232 29438 raft_consensus.cc:729] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3587b2902bc647269da683b5195eed6c, State: Initialized, Role: FOLLOWER
I20260810 18:50:06.642199 29438 consensus_queue.cc:260] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } } peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } }
I20260810 18:50:06.645538 29438 ts_tablet_manager.cc:1428] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c: Time spent starting tablet: real 0.010s user 0.002s sys 0.006s
I20260810 18:50:06.697775 29046 ts_tablet_manager-itest.cc:367] Electing peer 2...
I20260810 18:50:06.698366 29046 raft_consensus.cc:412] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 0 FOLLOWER]: Emulating election...
I20260810 18:50:06.698740 29046 raft_consensus.cc:3037] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:06.705149 29046 raft_consensus.cc:686] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 1 LEADER]: Becoming Leader. State: Replica: 8c7fc479c1ed4c37a3fc2914b5a6efda, State: Running, Role: LEADER
I20260810 18:50:06.705861 29046 consensus_queue.cc:237] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } } peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } }
I20260810 18:50:06.717993 29046 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260810 18:50:06.732937 29098 catalog_manager.cc:5495] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c7fc479c1ed4c37a3fc2914b5a6efda (127.28.93.130). New cstate: current_term: 1 leader_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:06.763866 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260810 18:50:06.870066 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260810 18:50:07.074955 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260810 18:50:07.153903 29251 raft_consensus.cc:3037] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.219185 29441 consensus_queue.cc:1035] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Connected to new peer: Peer: permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:07.378624 29046 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260810 18:50:07.378928 29046 raft_consensus.cc:412] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 0 FOLLOWER]: Emulating election...
I20260810 18:50:07.379204 29046 raft_consensus.cc:3037] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:07.415517 29046 raft_consensus.cc:686] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 1 LEADER]: Becoming Leader. State: Replica: 8c7fc479c1ed4c37a3fc2914b5a6efda, State: Running, Role: LEADER
I20260810 18:50:07.416290 29046 consensus_queue.cc:237] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:07.420827 29046 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260810 18:50:07.423839 29096 catalog_manager.cc:5495] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c7fc479c1ed4c37a3fc2914b5a6efda (127.28.93.130). New cstate: current_term: 1 leader_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:07.424646 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260810 18:50:07.528046 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260810 18:50:07.734615 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260810 18:50:08.035485 29251 raft_consensus.cc:3037] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:08.042035 29455 consensus_queue.cc:1035] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Connected to new peer: Peer: permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:08.045058 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260810 18:50:08.648811 29046 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260810 18:50:08.649066 29046 raft_consensus.cc:412] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 1 LEADER]: Emulating election...
I20260810 18:50:08.649333 29046 raft_consensus.cc:3032] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:08.649528 29046 raft_consensus.cc:729] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c7fc479c1ed4c37a3fc2914b5a6efda, State: Running, Role: LEADER
I20260810 18:50:08.649940 29046 consensus_queue.cc:260] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:08.650689 29046 raft_consensus.cc:3037] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:08.655002 29046 raft_consensus.cc:686] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 2 LEADER]: Becoming Leader. State: Replica: 8c7fc479c1ed4c37a3fc2914b5a6efda, State: Running, Role: LEADER
I20260810 18:50:08.655483 29046 consensus_queue.cc:237] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:08.656711 29046 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260810 18:50:08.660005 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260810 18:50:08.660081 29099 catalog_manager.cc:5495] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:08.764174 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260810 18:50:08.967336 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260810 18:50:09.140959 29249 raft_consensus.cc:3037] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:09.145821 29249 raft_consensus.cc:1261] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 2 FOLLOWER]: Refusing update from remote peer 8c7fc479c1ed4c37a3fc2914b5a6efda: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.146898 29477 consensus_queue.cc:1035] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Connected to new peer: Peer: permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:09.272025 29046 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260810 18:50:09.272322 29046 raft_consensus.cc:412] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [term 1 FOLLOWER]: Emulating election...
I20260810 18:50:09.272543 29046 raft_consensus.cc:3037] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:09.277621 29046 raft_consensus.cc:686] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [term 2 LEADER]: Becoming Leader. State: Replica: 3587b2902bc647269da683b5195eed6c, State: Running, Role: LEADER
I20260810 18:50:09.278301 29046 consensus_queue.cc:237] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } } peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } }
I20260810 18:50:09.280179 29046 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260810 18:50:09.284890 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260810 18:50:09.285110 29099 catalog_manager.cc:5495] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c reported cstate change: term changed from 1 to 2, leader changed from 8c7fc479c1ed4c37a3fc2914b5a6efda (127.28.93.130) to 3587b2902bc647269da683b5195eed6c (127.28.93.129). New cstate: current_term: 2 leader_uuid: "3587b2902bc647269da683b5195eed6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:09.390143 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260810 18:50:09.594874 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260810 18:50:09.602574 29249 raft_consensus.cc:1226] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [term 2 LEADER]: Rejecting Update request from peer 8c7fc479c1ed4c37a3fc2914b5a6efda for earlier term 1. Current term is 2. Ops: []
I20260810 18:50:09.604715 29477 consensus_queue.cc:1046] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:50:09.605789 29477 consensus_queue.cc:1225] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Peer 3587b2902bc647269da683b5195eed6c log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260810 18:50:09.606527 29494 raft_consensus.cc:3032] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:09.607065 29494 raft_consensus.cc:729] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c7fc479c1ed4c37a3fc2914b5a6efda, State: Running, Role: LEADER
I20260810 18:50:09.608301 29494 consensus_queue.cc:260] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } } peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } }
I20260810 18:50:09.610164 29494 raft_consensus.cc:3037] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:09.900466 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260810 18:50:09.907593 29376 raft_consensus.cc:1261] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 2 FOLLOWER]: Refusing update from remote peer 3587b2902bc647269da683b5195eed6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:09.910248 29502 consensus_queue.cc:1035] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260810 18:50:10.505676 29046 ts_tablet_manager-itest.cc:367] Electing peer 1...
I20260810 18:50:10.506002 29046 raft_consensus.cc:412] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 2 FOLLOWER]: Emulating election...
I20260810 18:50:10.506305 29046 raft_consensus.cc:3037] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:10.510921 29046 raft_consensus.cc:686] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 3 LEADER]: Becoming Leader. State: Replica: 3587b2902bc647269da683b5195eed6c, State: Running, Role: LEADER
I20260810 18:50:10.511512 29046 consensus_queue.cc:237] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:10.513562 29046 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260810 18:50:10.517166 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260810 18:50:10.517481 29098 catalog_manager.cc:5495] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c reported cstate change: term changed from 2 to 3, leader changed from 8c7fc479c1ed4c37a3fc2914b5a6efda (127.28.93.130) to 3587b2902bc647269da683b5195eed6c (127.28.93.129). New cstate: current_term: 3 leader_uuid: "3587b2902bc647269da683b5195eed6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:10.594240 29249 raft_consensus.cc:1226] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 3 LEADER]: Rejecting Update request from peer 8c7fc479c1ed4c37a3fc2914b5a6efda for earlier term 2. Current term is 3. Ops: []
I20260810 18:50:10.595266 29494 consensus_queue.cc:1046] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:10.596385 29494 consensus_queue.cc:1225] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Peer 3587b2902bc647269da683b5195eed6c log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260810 18:50:10.596735 29528 raft_consensus.cc:3032] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:10.596987 29528 raft_consensus.cc:729] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8c7fc479c1ed4c37a3fc2914b5a6efda, State: Running, Role: LEADER
I20260810 18:50:10.597406 29528 consensus_queue.cc:260] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:10.598047 29528 raft_consensus.cc:3037] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:10.620347 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260810 18:50:10.823608 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260810 18:50:11.080185 29376 raft_consensus.cc:1261] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 3 FOLLOWER]: Refusing update from remote peer 3587b2902bc647269da683b5195eed6c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:11.081423 29508 consensus_queue.cc:1035] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:11.127018 29046 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260810 18:50:11.127238 29046 raft_consensus.cc:412] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 3 FOLLOWER]: Emulating election...
I20260810 18:50:11.127416 29046 raft_consensus.cc:3037] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:11.131383 29046 raft_consensus.cc:686] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 4 LEADER]: Becoming Leader. State: Replica: 8c7fc479c1ed4c37a3fc2914b5a6efda, State: Running, Role: LEADER
I20260810 18:50:11.132032 29046 consensus_queue.cc:237] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:11.133430 29046 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260810 18:50:11.136801 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260810 18:50:11.136505 29099 catalog_manager.cc:5495] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda reported cstate change: term changed from 3 to 4, leader changed from 3587b2902bc647269da683b5195eed6c (127.28.93.129) to 8c7fc479c1ed4c37a3fc2914b5a6efda (127.28.93.130). New cstate: current_term: 4 leader_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:11.240211 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260810 18:50:11.444981 29046 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260810 18:50:11.643332 29249 raft_consensus.cc:3032] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:50:11.643692 29249 raft_consensus.cc:729] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3587b2902bc647269da683b5195eed6c, State: Running, Role: LEADER
I20260810 18:50:11.644214 29249 consensus_queue.cc:260] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 } } peers { permanent_uuid: "8c7fc479c1ed4c37a3fc2914b5a6efda" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 43759 } }
I20260810 18:50:11.644904 29249 raft_consensus.cc:3037] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:11.650615 29249 raft_consensus.cc:1261] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 4 FOLLOWER]: Refusing update from remote peer 8c7fc479c1ed4c37a3fc2914b5a6efda: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260810 18:50:11.652148 29545 consensus_queue.cc:1035] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [LEADER]: Connected to new peer: Peer: permanent_uuid: "3587b2902bc647269da683b5195eed6c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:50:11.951222 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:11.969959 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:11.970580 29046 tablet_replica.cc:330] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c: stopping tablet replica
I20260810 18:50:11.971139 29046 raft_consensus.cc:2229] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [term 2 LEADER]: Raft consensus shutting down.
I20260810 18:50:11.971778 29046 raft_consensus.cc:2258] T 85cb4e5c60a6430a98c8783d0233e531 P 3587b2902bc647269da683b5195eed6c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:11.973613 29046 tablet_replica.cc:330] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c: stopping tablet replica
I20260810 18:50:11.974006 29046 raft_consensus.cc:2229] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:11.974332 29046 raft_consensus.cc:2258] T 3ebdcb16d19c413d945efe3dd98e527f P 3587b2902bc647269da683b5195eed6c [term 4 FOLLOWER]: Raft consensus is shut down!
W20260810 18:50:12.065577 29320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.93.129:41077: connect: Connection refused (error 111)
W20260810 18:50:12.068940 29320 consensus_peers.cc:487] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda -> Peer 3587b2902bc647269da683b5195eed6c (127.28.93.129:41077): Couldn't send request to peer 3587b2902bc647269da683b5195eed6c. Status: Network error: Client connection negotiation failed: client connection to 127.28.93.129:41077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:12.069772 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:12.078059 29046 tablet_server.cc:178] TabletServer@127.28.93.130:0 shutting down...
I20260810 18:50:12.093186 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:12.093662 29046 tablet_replica.cc:330] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda: stopping tablet replica
I20260810 18:50:12.094105 29046 raft_consensus.cc:2229] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 2 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:12.094463 29046 raft_consensus.cc:2258] T 85cb4e5c60a6430a98c8783d0233e531 P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:12.095803 29046 tablet_replica.cc:330] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda: stopping tablet replica
I20260810 18:50:12.096215 29046 raft_consensus.cc:2229] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 4 LEADER]: Raft consensus shutting down.
I20260810 18:50:12.096702 29046 raft_consensus.cc:2258] T 3ebdcb16d19c413d945efe3dd98e527f P 8c7fc479c1ed4c37a3fc2914b5a6efda [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:12.114682 29046 tablet_server.cc:194] TabletServer@127.28.93.130:0 shutdown complete.
I20260810 18:50:12.122356 29046 master.cc:537] Master@127.28.93.190:44219 shutting down...
I20260810 18:50:12.271317 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:12.271821 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:12.272137 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P a9410b30c55f406ba5159d2e8242b06e: stopping tablet replica
I20260810 18:50:12.288923 29046 master.cc:552] Master@127.28.93.190:44219 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (8356 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20260810 18:50:12.313304 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:36133
I20260810 18:50:12.314217 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:12.318859 29561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:12.319933 29562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:12.321228 29565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:12.323189 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:12.323832 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:12.323998 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:12.324141 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387812324129 us; error 0 us; skew 500 ppm
I20260810 18:50:12.324618 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:12.330796 29046 webserver.cc:460] Webserver started at http://127.28.93.190:33817/ using document root <none> and password file <none>
I20260810 18:50:12.331223 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:12.331395 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:12.331634 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:12.332692 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "fa2a99f407da4d8a9ae715fd1ce87432"
format_stamp: "Formatted at 2026-08-10 18:50:12 on dist-test-slave-nj21"
I20260810 18:50:12.336774 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:50:12.339581 29580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:12.340268 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:12.340515 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "fa2a99f407da4d8a9ae715fd1ce87432"
format_stamp: "Formatted at 2026-08-10 18:50:12 on dist-test-slave-nj21"
I20260810 18:50:12.340759 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:12.351495 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:12.352489 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:12.384508 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:36133
I20260810 18:50:12.384588 29648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:36133 every 8 connection(s)
I20260810 18:50:12.388048 29651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:12.396409 29651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432: Bootstrap starting.
I20260810 18:50:12.399955 29651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:12.403299 29651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432: No bootstrap required, opened a new log
I20260810 18:50:12.404937 29651 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2a99f407da4d8a9ae715fd1ce87432" member_type: VOTER }
I20260810 18:50:12.405293 29651 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:12.405470 29651 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa2a99f407da4d8a9ae715fd1ce87432, State: Initialized, Role: FOLLOWER
I20260810 18:50:12.406056 29651 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2a99f407da4d8a9ae715fd1ce87432" member_type: VOTER }
I20260810 18:50:12.406508 29651 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:12.406690 29651 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:12.406894 29651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:12.410463 29651 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2a99f407da4d8a9ae715fd1ce87432" member_type: VOTER }
I20260810 18:50:12.410900 29651 leader_election.cc:302] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa2a99f407da4d8a9ae715fd1ce87432; no voters:
I20260810 18:50:12.411919 29651 leader_election.cc:288] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:12.412278 29656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:12.413540 29656 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 1 LEADER]: Becoming Leader. State: Replica: fa2a99f407da4d8a9ae715fd1ce87432, State: Running, Role: LEADER
I20260810 18:50:12.414042 29656 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2a99f407da4d8a9ae715fd1ce87432" member_type: VOTER }
I20260810 18:50:12.414497 29651 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:12.416563 29658 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa2a99f407da4d8a9ae715fd1ce87432. Latest consensus state: current_term: 1 leader_uuid: "fa2a99f407da4d8a9ae715fd1ce87432" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2a99f407da4d8a9ae715fd1ce87432" member_type: VOTER } }
I20260810 18:50:12.416555 29657 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa2a99f407da4d8a9ae715fd1ce87432" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2a99f407da4d8a9ae715fd1ce87432" member_type: VOTER } }
I20260810 18:50:12.417145 29658 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:12.417161 29657 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:12.420763 29670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:12.427438 29670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:12.428344 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:12.434798 29670 catalog_manager.cc:1347] Generated new cluster ID: 88a17378540a423e8d7caa8e52b0830b
I20260810 18:50:12.435036 29670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:12.459349 29670 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:12.460835 29670 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:12.473304 29670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432: Generated new TSK 0
I20260810 18:50:12.473884 29670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:12.494076 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:12.499402 29688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:12.500550 29689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:12.501461 29695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:12.502420 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:12.503185 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:12.503362 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:12.503481 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387812503473 us; error 0 us; skew 500 ppm
I20260810 18:50:12.503947 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:12.505993 29046 webserver.cc:460] Webserver started at http://127.28.93.129:36807/ using document root <none> and password file <none>
I20260810 18:50:12.506420 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:12.506575 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:12.506771 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:12.508059 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "0eece3ed31dd4ca4b21fd3c17b747e8a"
format_stamp: "Formatted at 2026-08-10 18:50:12 on dist-test-slave-nj21"
I20260810 18:50:12.512203 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:50:12.515080 29703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:12.515758 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:12.515991 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "0eece3ed31dd4ca4b21fd3c17b747e8a"
format_stamp: "Formatted at 2026-08-10 18:50:12 on dist-test-slave-nj21"
I20260810 18:50:12.516224 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:12.545753 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:12.546682 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:12.547963 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:12.550110 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:12.550274 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:12.550520 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:12.550657 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:12.588783 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:46703
I20260810 18:50:12.588827 29791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:46703 every 8 connection(s)
I20260810 18:50:12.596235 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:12.607651 29796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:12.610083 29797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:12.616359 29799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:12.616930 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:12.617985 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:12.618227 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:12.618435 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387812618419 us; error 0 us; skew 500 ppm
I20260810 18:50:12.619099 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:12.622308 29046 webserver.cc:460] Webserver started at http://127.28.93.130:46343/ using document root <none> and password file <none>
I20260810 18:50:12.622917 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:12.623158 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:12.623468 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:12.624486 29792 heartbeater.cc:347] Connected to a master server at 127.28.93.190:36133
I20260810 18:50:12.624837 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/ts-1-root/instance:
uuid: "cec0e7be7d8e4b1a8102ae9ea612f88f"
format_stamp: "Formatted at 2026-08-10 18:50:12 on dist-test-slave-nj21"
I20260810 18:50:12.624956 29792 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:12.626237 29792 heartbeater.cc:511] Master 127.28.93.190:36133 requested a full tablet report, sending...
I20260810 18:50:12.629629 29598 ts_manager.cc:194] Registered new tserver with Master: 0eece3ed31dd4ca4b21fd3c17b747e8a (127.28.93.129:46703)
I20260810 18:50:12.630911 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 18:50:12.633098 29598 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46646
I20260810 18:50:12.635563 29810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:12.636693 29046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 18:50:12.636945 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/ts-1-root
uuid: "cec0e7be7d8e4b1a8102ae9ea612f88f"
format_stamp: "Formatted at 2026-08-10 18:50:12 on dist-test-slave-nj21"
I20260810 18:50:12.637199 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786387803638833-29046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:12.655633 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:12.657064 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:12.659718 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:12.663309 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:12.663482 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:12.663726 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:12.663863 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:12.738529 29792 heartbeater.cc:503] Master 127.28.93.190:36133 was elected leader, sending a full tablet report...
I20260810 18:50:12.764734 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.130:35537
I20260810 18:50:12.764807 29910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.130:35537 every 8 connection(s)
I20260810 18:50:12.778467 29911 heartbeater.cc:347] Connected to a master server at 127.28.93.190:36133
I20260810 18:50:12.778793 29911 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:12.779420 29911 heartbeater.cc:511] Master 127.28.93.190:36133 requested a full tablet report, sending...
I20260810 18:50:12.781141 29598 ts_manager.cc:194] Registered new tserver with Master: cec0e7be7d8e4b1a8102ae9ea612f88f (127.28.93.130:35537)
I20260810 18:50:12.782121 29046 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.014231701s
I20260810 18:50:12.782833 29598 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46652
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:421: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20260810 18:50:12.803134 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:12.816839 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:12.831902 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:12.838164 29046 tablet_server.cc:178] TabletServer@127.28.93.130:0 shutting down...
I20260810 18:50:12.852344 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:12.867271 29046 tablet_server.cc:194] TabletServer@127.28.93.130:0 shutdown complete.
I20260810 18:50:12.873332 29046 master.cc:537] Master@127.28.93.190:36133 shutting down...
I20260810 18:50:12.885494 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:12.885928 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:12.886245 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P fa2a99f407da4d8a9ae715fd1ce87432: stopping tablet replica
I20260810 18:50:12.903605 29046 master.cc:552] Master@127.28.93.190:36133 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (609 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20260810 18:50:12.922986 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:35509
I20260810 18:50:12.923935 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:12.928187 29934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:12.929337 29938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:12.930531 29941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:12.931568 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:12.932266 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:12.932428 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:12.932535 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387812932528 us; error 0 us; skew 500 ppm
I20260810 18:50:12.932946 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:12.935038 29046 webserver.cc:460] Webserver started at http://127.28.93.190:42075/ using document root <none> and password file <none>
I20260810 18:50:12.935431 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:12.935570 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:12.935895 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:12.936854 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "cf819c6fa3bf47e7b99ff4915c8af8a8"
format_stamp: "Formatted at 2026-08-10 18:50:12 on dist-test-slave-nj21"
I20260810 18:50:12.940783 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:50:12.943696 29948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:12.944403 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:12.944624 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "cf819c6fa3bf47e7b99ff4915c8af8a8"
format_stamp: "Formatted at 2026-08-10 18:50:12 on dist-test-slave-nj21"
I20260810 18:50:12.944852 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:12.971488 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:12.972455 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:13.004769 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:35509
I20260810 18:50:13.004849 30032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:35509 every 8 connection(s)
I20260810 18:50:13.008224 30035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:13.016659 30035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8: Bootstrap starting.
I20260810 18:50:13.020213 30035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:13.024044 30035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8: No bootstrap required, opened a new log
I20260810 18:50:13.026016 30035 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf819c6fa3bf47e7b99ff4915c8af8a8" member_type: VOTER }
I20260810 18:50:13.026381 30035 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:13.026549 30035 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf819c6fa3bf47e7b99ff4915c8af8a8, State: Initialized, Role: FOLLOWER
I20260810 18:50:13.026979 30035 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf819c6fa3bf47e7b99ff4915c8af8a8" member_type: VOTER }
I20260810 18:50:13.027357 30035 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:13.027530 30035 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:13.027767 30035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:13.031189 30035 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf819c6fa3bf47e7b99ff4915c8af8a8" member_type: VOTER }
I20260810 18:50:13.031611 30035 leader_election.cc:302] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf819c6fa3bf47e7b99ff4915c8af8a8; no voters:
I20260810 18:50:13.032630 30035 leader_election.cc:288] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:13.032876 30041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:13.034060 30041 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 1 LEADER]: Becoming Leader. State: Replica: cf819c6fa3bf47e7b99ff4915c8af8a8, State: Running, Role: LEADER
I20260810 18:50:13.034655 30041 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf819c6fa3bf47e7b99ff4915c8af8a8" member_type: VOTER }
I20260810 18:50:13.035372 30035 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:13.037146 30042 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf819c6fa3bf47e7b99ff4915c8af8a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf819c6fa3bf47e7b99ff4915c8af8a8" member_type: VOTER } }
I20260810 18:50:13.037179 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf819c6fa3bf47e7b99ff4915c8af8a8. Latest consensus state: current_term: 1 leader_uuid: "cf819c6fa3bf47e7b99ff4915c8af8a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf819c6fa3bf47e7b99ff4915c8af8a8" member_type: VOTER } }
I20260810 18:50:13.037914 30042 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:13.037925 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:13.046732 30049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:13.053047 30049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:13.053905 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:13.060645 30049 catalog_manager.cc:1347] Generated new cluster ID: 76572341471747a89c3f29c2f601af12
I20260810 18:50:13.060846 30049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:13.081682 30049 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:13.083122 30049 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:13.095548 30049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8: Generated new TSK 0
I20260810 18:50:13.096204 30049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:13.119886 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:13.125173 30070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:13.126191 30074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:13.127148 30077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:13.127534 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:13.128297 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:13.128500 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:13.128649 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387813128636 us; error 0 us; skew 500 ppm
I20260810 18:50:13.129130 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:13.131269 29046 webserver.cc:460] Webserver started at http://127.28.93.129:34173/ using document root <none> and password file <none>
I20260810 18:50:13.131772 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:13.131940 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:13.132192 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:13.133194 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "070adda191fb46249934b2624cb8d369"
format_stamp: "Formatted at 2026-08-10 18:50:13 on dist-test-slave-nj21"
I20260810 18:50:13.137471 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:50:13.140292 30082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:13.140933 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:13.141175 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "070adda191fb46249934b2624cb8d369"
format_stamp: "Formatted at 2026-08-10 18:50:13 on dist-test-slave-nj21"
I20260810 18:50:13.141443 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:13.154160 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:13.155102 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:13.156417 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:13.158506 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:13.158684 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:13.158895 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:13.159032 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:13.192806 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:44691
I20260810 18:50:13.192898 30182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:44691 every 8 connection(s)
I20260810 18:50:13.196870 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:13.203387 30195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:13.205008 30197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:13.207463 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:13.207984 30201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:13.208952 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:13.209151 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:13.209299 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387813209285 us; error 0 us; skew 500 ppm
I20260810 18:50:13.209889 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:13.209988 30184 heartbeater.cc:347] Connected to a master server at 127.28.93.190:35509
I20260810 18:50:13.210330 30184 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:13.211026 30184 heartbeater.cc:511] Master 127.28.93.190:35509 requested a full tablet report, sending...
I20260810 18:50:13.212381 29046 webserver.cc:460] Webserver started at http://127.28.93.130:44805/ using document root <none> and password file <none>
I20260810 18:50:13.212818 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:13.212831 29977 ts_manager.cc:194] Registered new tserver with Master: 070adda191fb46249934b2624cb8d369 (127.28.93.129:44691)
I20260810 18:50:13.213042 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:13.213335 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:13.214310 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/ts-1-root/instance:
uuid: "3ba4bfed39364538b7d185570953fb0d"
format_stamp: "Formatted at 2026-08-10 18:50:13 on dist-test-slave-nj21"
I20260810 18:50:13.214659 29977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51746
I20260810 18:50:13.218262 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:50:13.221143 30210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:13.221786 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:50:13.222087 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/ts-1-root
uuid: "3ba4bfed39364538b7d185570953fb0d"
format_stamp: "Formatted at 2026-08-10 18:50:13 on dist-test-slave-nj21"
I20260810 18:50:13.222404 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786387803638833-29046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:13.243252 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:13.244202 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:13.245462 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:13.247547 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:13.247752 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:13.247974 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:13.248104 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:13.285897 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.130:34975
I20260810 18:50:13.285977 30305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.130:34975 every 8 connection(s)
I20260810 18:50:13.299606 30309 heartbeater.cc:347] Connected to a master server at 127.28.93.190:35509
I20260810 18:50:13.299952 30309 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:13.300618 30309 heartbeater.cc:511] Master 127.28.93.190:35509 requested a full tablet report, sending...
I20260810 18:50:13.302273 29977 ts_manager.cc:194] Registered new tserver with Master: 3ba4bfed39364538b7d185570953fb0d (127.28.93.130:34975)
I20260810 18:50:13.302588 29046 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.013824004s
I20260810 18:50:13.303822 29977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51750
I20260810 18:50:13.317093 30184 heartbeater.cc:503] Master 127.28.93.190:35509 was elected leader, sending a full tablet report...
I20260810 18:50:13.324411 29977 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:50:13.325691 29977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260810 18:50:13.354537 30124 tablet_service.cc:1467] Processing CreateTablet for tablet 4544b214fa08401db5597ab8af8384a1 (DEFAULT_TABLE table=test-table [id=05244e492e864a1ba69021af6b99411f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:13.355443 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4544b214fa08401db5597ab8af8384a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:13.358157 30253 tablet_service.cc:1467] Processing CreateTablet for tablet 4544b214fa08401db5597ab8af8384a1 (DEFAULT_TABLE table=test-table [id=05244e492e864a1ba69021af6b99411f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:13.359261 30253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4544b214fa08401db5597ab8af8384a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:13.367400 30333 tablet_bootstrap.cc:492] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369: Bootstrap starting.
I20260810 18:50:13.369395 30335 tablet_bootstrap.cc:492] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d: Bootstrap starting.
I20260810 18:50:13.372608 30333 tablet_bootstrap.cc:654] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:13.374702 30335 tablet_bootstrap.cc:654] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:13.377034 30333 tablet_bootstrap.cc:492] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369: No bootstrap required, opened a new log
I20260810 18:50:13.377444 30333 ts_tablet_manager.cc:1397] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 18:50:13.379446 30335 tablet_bootstrap.cc:492] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d: No bootstrap required, opened a new log
I20260810 18:50:13.379364 30333 raft_consensus.cc:348] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070adda191fb46249934b2624cb8d369" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44691 } } peers { permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 } }
I20260810 18:50:13.379902 30335 ts_tablet_manager.cc:1397] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260810 18:50:13.379972 30333 raft_consensus.cc:729] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070adda191fb46249934b2624cb8d369, State: Initialized, Role: FOLLOWER
I20260810 18:50:13.380493 30333 consensus_queue.cc:260] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070adda191fb46249934b2624cb8d369" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44691 } } peers { permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 } }
I20260810 18:50:13.381636 30335 raft_consensus.cc:348] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070adda191fb46249934b2624cb8d369" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44691 } } peers { permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 } }
I20260810 18:50:13.382090 30335 raft_consensus.cc:729] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba4bfed39364538b7d185570953fb0d, State: Initialized, Role: FOLLOWER
I20260810 18:50:13.382507 30335 consensus_queue.cc:260] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070adda191fb46249934b2624cb8d369" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44691 } } peers { permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 } }
I20260810 18:50:13.383649 30333 ts_tablet_manager.cc:1428] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260810 18:50:13.385258 30309 heartbeater.cc:503] Master 127.28.93.190:35509 was elected leader, sending a full tablet report...
I20260810 18:50:13.385531 30335 ts_tablet_manager.cc:1428] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260810 18:50:13.416662 29046 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260810 18:50:13.416884 29046 raft_consensus.cc:412] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 0 FOLLOWER]: Emulating election...
I20260810 18:50:13.417079 29046 raft_consensus.cc:3037] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:13.420737 29046 raft_consensus.cc:686] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 1 LEADER]: Becoming Leader. State: Replica: 070adda191fb46249934b2624cb8d369, State: Running, Role: LEADER
W20260810 18:50:13.421021 30184 heartbeater.cc:644] Failed to heartbeat to 127.28.93.190:35509 (0 consecutive failures): IO error: failing all heartbeats for tests
I20260810 18:50:13.421314 29046 consensus_queue.cc:237] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070adda191fb46249934b2624cb8d369" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44691 } } peers { permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 } }
I20260810 18:50:13.423982 29046 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260810 18:50:13.442869 29046 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260810 18:50:13.546375 29046 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260810 18:50:13.749903 29046 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260810 18:50:14.031438 30264 raft_consensus.cc:3037] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:14.038151 30347 consensus_queue.cc:1035] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:14.054929 29046 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260810 18:50:14.055132 29046 raft_consensus.cc:412] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 1 LEADER]: Emulating election...
I20260810 18:50:14.055316 29046 raft_consensus.cc:3032] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:14.055470 29046 raft_consensus.cc:729] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 070adda191fb46249934b2624cb8d369, State: Running, Role: LEADER
I20260810 18:50:14.055871 29046 consensus_queue.cc:260] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070adda191fb46249934b2624cb8d369" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44691 } } peers { permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 } }
I20260810 18:50:14.056486 29046 raft_consensus.cc:3037] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:14.060612 29046 raft_consensus.cc:686] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 2 LEADER]: Becoming Leader. State: Replica: 070adda191fb46249934b2624cb8d369, State: Running, Role: LEADER
I20260810 18:50:14.061107 29046 consensus_queue.cc:237] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070adda191fb46249934b2624cb8d369" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44691 } } peers { permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 } }
I20260810 18:50:14.062316 29046 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260810 18:50:14.065238 29046 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260810 18:50:14.168840 29046 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260810 18:50:14.372483 29046 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260810 18:50:14.506661 30264 raft_consensus.cc:3037] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:14.511253 30264 raft_consensus.cc:1261] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d [term 2 FOLLOWER]: Refusing update from remote peer 070adda191fb46249934b2624cb8d369: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:14.512458 30352 consensus_queue.cc:1035] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:14.677162 29046 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260810 18:50:14.677405 29046 raft_consensus.cc:412] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 2 LEADER]: Emulating election...
I20260810 18:50:14.677606 29046 raft_consensus.cc:3032] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:14.677783 29046 raft_consensus.cc:729] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 070adda191fb46249934b2624cb8d369, State: Running, Role: LEADER
I20260810 18:50:14.678181 29046 consensus_queue.cc:260] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070adda191fb46249934b2624cb8d369" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44691 } } peers { permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 } }
I20260810 18:50:14.678802 29046 raft_consensus.cc:3037] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:14.682547 29046 raft_consensus.cc:686] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 3 LEADER]: Becoming Leader. State: Replica: 070adda191fb46249934b2624cb8d369, State: Running, Role: LEADER
I20260810 18:50:14.683044 29046 consensus_queue.cc:237] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "070adda191fb46249934b2624cb8d369" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44691 } } peers { permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 } }
I20260810 18:50:14.684237 29046 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260810 18:50:14.687125 29046 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260810 18:50:14.790829 29046 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260810 18:50:14.994563 29046 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260810 18:50:15.178588 30264 raft_consensus.cc:3037] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:15.183130 30264 raft_consensus.cc:1261] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d [term 3 FOLLOWER]: Refusing update from remote peer 070adda191fb46249934b2624cb8d369: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:15.184458 30357 consensus_queue.cc:1035] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba4bfed39364538b7d185570953fb0d" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:15.301218 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:15.321193 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:15.321782 29046 tablet_replica.cc:330] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369: stopping tablet replica
I20260810 18:50:15.322301 29046 raft_consensus.cc:2229] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:50:15.322867 29046 raft_consensus.cc:2258] T 4544b214fa08401db5597ab8af8384a1 P 070adda191fb46249934b2624cb8d369 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:15.330978 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:15.339138 29046 tablet_server.cc:178] TabletServer@127.28.93.130:0 shutting down...
I20260810 18:50:15.356320 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:15.356819 29046 tablet_replica.cc:330] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d: stopping tablet replica
I20260810 18:50:15.357246 29046 raft_consensus.cc:2229] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:15.357604 29046 raft_consensus.cc:2258] T 4544b214fa08401db5597ab8af8384a1 P 3ba4bfed39364538b7d185570953fb0d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:15.375460 29046 tablet_server.cc:194] TabletServer@127.28.93.130:0 shutdown complete.
I20260810 18:50:15.383074 29046 master.cc:537] Master@127.28.93.190:35509 shutting down...
I20260810 18:50:15.398587 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:15.399096 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:15.399351 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P cf819c6fa3bf47e7b99ff4915c8af8a8: stopping tablet replica
I20260810 18:50:15.415504 29046 master.cc:552] Master@127.28.93.190:35509 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2519 ms)
[----------] 3 tests from LeadershipChangeReportingTest (11485 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20260810 18:50:15.443121 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:43307
I20260810 18:50:15.444156 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:15.449150 30376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:15.449954 30377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:15.452601 30382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:15.452682 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:15.453501 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:15.453679 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:15.453786 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387815453779 us; error 0 us; skew 500 ppm
I20260810 18:50:15.454206 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:15.456497 29046 webserver.cc:460] Webserver started at http://127.28.93.190:35441/ using document root <none> and password file <none>
I20260810 18:50:15.456923 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:15.457067 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:15.457284 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:15.458336 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "8e0857e664c14bcb8abd8ef3f5359d23"
format_stamp: "Formatted at 2026-08-10 18:50:15 on dist-test-slave-nj21"
I20260810 18:50:15.462456 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:50:15.465482 30389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:15.466130 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:50:15.466356 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "8e0857e664c14bcb8abd8ef3f5359d23"
format_stamp: "Formatted at 2026-08-10 18:50:15 on dist-test-slave-nj21"
I20260810 18:50:15.466583 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:15.498229 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:15.499559 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:15.540506 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:43307
I20260810 18:50:15.540613 30471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:43307 every 8 connection(s)
I20260810 18:50:15.544191 30472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:15.553550 30472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23: Bootstrap starting.
I20260810 18:50:15.557647 30472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:15.561465 30472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23: No bootstrap required, opened a new log
I20260810 18:50:15.563293 30472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0857e664c14bcb8abd8ef3f5359d23" member_type: VOTER }
I20260810 18:50:15.563654 30472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:15.563886 30472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0857e664c14bcb8abd8ef3f5359d23, State: Initialized, Role: FOLLOWER
I20260810 18:50:15.564388 30472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0857e664c14bcb8abd8ef3f5359d23" member_type: VOTER }
I20260810 18:50:15.564795 30472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:15.564999 30472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:15.565227 30472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:15.569270 30472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0857e664c14bcb8abd8ef3f5359d23" member_type: VOTER }
I20260810 18:50:15.569744 30472 leader_election.cc:302] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e0857e664c14bcb8abd8ef3f5359d23; no voters:
I20260810 18:50:15.570742 30472 leader_election.cc:288] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:15.571101 30478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:15.572328 30478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 1 LEADER]: Becoming Leader. State: Replica: 8e0857e664c14bcb8abd8ef3f5359d23, State: Running, Role: LEADER
I20260810 18:50:15.572934 30478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0857e664c14bcb8abd8ef3f5359d23" member_type: VOTER }
I20260810 18:50:15.573493 30472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:15.575644 30480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e0857e664c14bcb8abd8ef3f5359d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0857e664c14bcb8abd8ef3f5359d23" member_type: VOTER } }
I20260810 18:50:15.575654 30482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e0857e664c14bcb8abd8ef3f5359d23. Latest consensus state: current_term: 1 leader_uuid: "8e0857e664c14bcb8abd8ef3f5359d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0857e664c14bcb8abd8ef3f5359d23" member_type: VOTER } }
I20260810 18:50:15.576337 30480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:15.576387 30482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:15.584841 30489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:15.590579 30489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:15.591522 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:15.598567 30489 catalog_manager.cc:1347] Generated new cluster ID: d76d557946ea453badcc07d182556c31
I20260810 18:50:15.598834 30489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:15.624729 30489 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:15.626292 30489 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:15.647321 30489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23: Generated new TSK 0
I20260810 18:50:15.647979 30489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:15.657600 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:15.663007 30508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:15.663939 30509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:15.666078 30511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:15.666167 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:15.667074 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:15.667245 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:15.667409 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387815667394 us; error 0 us; skew 500 ppm
I20260810 18:50:15.668004 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:15.670200 29046 webserver.cc:460] Webserver started at http://127.28.93.129:40627/ using document root <none> and password file <none>
I20260810 18:50:15.670630 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:15.670869 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:15.671213 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:15.672497 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "486704a2d6f04d019f30ce822842aedf"
format_stamp: "Formatted at 2026-08-10 18:50:15 on dist-test-slave-nj21"
I20260810 18:50:15.679862 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260810 18:50:15.686574 30520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:15.687577 29046 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260810 18:50:15.687937 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "486704a2d6f04d019f30ce822842aedf"
format_stamp: "Formatted at 2026-08-10 18:50:15 on dist-test-slave-nj21"
I20260810 18:50:15.688308 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:15.719923 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:15.721047 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:15.722494 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:15.724824 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:15.725015 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:15.725224 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:15.725373 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:15.762712 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:46741
I20260810 18:50:15.762795 30607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:46741 every 8 connection(s)
I20260810 18:50:15.767028 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:15.773552 30614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:15.774210 30617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:15.777240 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:15.777194 30622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:15.778219 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:15.778419 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:15.778602 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387815778587 us; error 0 us; skew 500 ppm
I20260810 18:50:15.779250 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:15.781246 30608 heartbeater.cc:347] Connected to a master server at 127.28.93.190:43307
I20260810 18:50:15.781601 30608 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:15.781942 29046 webserver.cc:460] Webserver started at http://127.28.93.130:44341/ using document root <none> and password file <none>
I20260810 18:50:15.782294 30608 heartbeater.cc:511] Master 127.28.93.190:43307 requested a full tablet report, sending...
I20260810 18:50:15.782404 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:15.782656 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:15.782915 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:15.784222 30420 ts_manager.cc:194] Registered new tserver with Master: 486704a2d6f04d019f30ce822842aedf (127.28.93.129:46741)
I20260810 18:50:15.784250 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-1-root/instance:
uuid: "34f3bcaa02664eb2ad6f3a1db136739e"
format_stamp: "Formatted at 2026-08-10 18:50:15 on dist-test-slave-nj21"
I20260810 18:50:15.786070 30420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59684
I20260810 18:50:15.788973 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:50:15.792270 30631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:15.793017 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260810 18:50:15.793256 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-1-root
uuid: "34f3bcaa02664eb2ad6f3a1db136739e"
format_stamp: "Formatted at 2026-08-10 18:50:15 on dist-test-slave-nj21"
I20260810 18:50:15.793507 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:15.831187 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:15.832460 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:15.833811 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:15.835991 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:15.836184 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:15.836406 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:15.836551 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:15.875384 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.130:33523
I20260810 18:50:15.875463 30726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.130:33523 every 8 connection(s)
I20260810 18:50:15.879962 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:15.886135 30730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:15.888238 30732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:15.890259 30734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:15.890228 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:15.891690 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:15.891966 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:15.892167 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387815892149 us; error 0 us; skew 500 ppm
I20260810 18:50:15.892869 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:15.893381 30727 heartbeater.cc:347] Connected to a master server at 127.28.93.190:43307
I20260810 18:50:15.893718 30727 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:15.894523 30727 heartbeater.cc:511] Master 127.28.93.190:43307 requested a full tablet report, sending...
I20260810 18:50:15.895479 29046 webserver.cc:460] Webserver started at http://127.28.93.131:38271/ using document root <none> and password file <none>
I20260810 18:50:15.896025 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:15.896200 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:15.896432 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:15.896561 30420 ts_manager.cc:194] Registered new tserver with Master: 34f3bcaa02664eb2ad6f3a1db136739e (127.28.93.130:33523)
I20260810 18:50:15.897519 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-2-root/instance:
uuid: "fb9aa4200f654b6786655d8422b3bba5"
format_stamp: "Formatted at 2026-08-10 18:50:15 on dist-test-slave-nj21"
I20260810 18:50:15.898103 30420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59692
I20260810 18:50:15.901772 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:50:15.904776 30741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:15.905440 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:15.905691 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-2-root
uuid: "fb9aa4200f654b6786655d8422b3bba5"
format_stamp: "Formatted at 2026-08-10 18:50:15 on dist-test-slave-nj21"
I20260810 18:50:15.905946 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786387803638833-29046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:15.927829 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:15.928952 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:15.930261 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:15.932325 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:15.932514 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:15.932722 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:15.932860 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:15.997767 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.131:38381
I20260810 18:50:15.997854 30846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.131:38381 every 8 connection(s)
I20260810 18:50:16.013478 30847 heartbeater.cc:347] Connected to a master server at 127.28.93.190:43307
I20260810 18:50:16.013784 30847 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:16.014408 30847 heartbeater.cc:511] Master 127.28.93.190:43307 requested a full tablet report, sending...
I20260810 18:50:16.016055 30420 ts_manager.cc:194] Registered new tserver with Master: fb9aa4200f654b6786655d8422b3bba5 (127.28.93.131:38381)
I20260810 18:50:16.016964 29046 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015393972s
I20260810 18:50:16.017520 30420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59700
I20260810 18:50:16.041826 30420 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260810 18:50:16.043409 30420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:50:16.084681 30561 tablet_service.cc:1467] Processing CreateTablet for tablet bab69f775f9e452ba812bc35e127fe1e (DEFAULT_TABLE table=test-table [id=0e212d515b824395b0a6a79684214ea8]), partition=
I20260810 18:50:16.085774 30561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab69f775f9e452ba812bc35e127fe1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:16.089111 30673 tablet_service.cc:1467] Processing CreateTablet for tablet bab69f775f9e452ba812bc35e127fe1e (DEFAULT_TABLE table=test-table [id=0e212d515b824395b0a6a79684214ea8]), partition=
I20260810 18:50:16.089228 30784 tablet_service.cc:1467] Processing CreateTablet for tablet bab69f775f9e452ba812bc35e127fe1e (DEFAULT_TABLE table=test-table [id=0e212d515b824395b0a6a79684214ea8]), partition=
I20260810 18:50:16.090041 30784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab69f775f9e452ba812bc35e127fe1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:16.090168 30673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab69f775f9e452ba812bc35e127fe1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:16.102039 30874 tablet_bootstrap.cc:492] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5: Bootstrap starting.
I20260810 18:50:16.103295 30873 tablet_bootstrap.cc:492] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf: Bootstrap starting.
I20260810 18:50:16.104983 30876 tablet_bootstrap.cc:492] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e: Bootstrap starting.
I20260810 18:50:16.107142 30874 tablet_bootstrap.cc:654] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:16.110353 30873 tablet_bootstrap.cc:654] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:16.112004 30876 tablet_bootstrap.cc:654] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:16.113044 30874 tablet_bootstrap.cc:492] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5: No bootstrap required, opened a new log
I20260810 18:50:16.113402 30874 ts_tablet_manager.cc:1397] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260810 18:50:16.115934 30874 raft_consensus.cc:348] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:16.116422 30874 raft_consensus.cc:374] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:16.116641 30874 raft_consensus.cc:729] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9aa4200f654b6786655d8422b3bba5, State: Initialized, Role: FOLLOWER
I20260810 18:50:16.117256 30873 tablet_bootstrap.cc:492] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf: No bootstrap required, opened a new log
I20260810 18:50:16.117727 30873 ts_tablet_manager.cc:1397] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260810 18:50:16.117522 30874 consensus_queue.cc:260] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:16.120440 30847 heartbeater.cc:503] Master 127.28.93.190:43307 was elected leader, sending a full tablet report...
I20260810 18:50:16.120396 30873 raft_consensus.cc:348] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:16.121174 30874 ts_tablet_manager.cc:1428] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 18:50:16.121208 30873 raft_consensus.cc:374] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:16.121587 30873 raft_consensus.cc:729] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486704a2d6f04d019f30ce822842aedf, State: Initialized, Role: FOLLOWER
I20260810 18:50:16.122246 30873 consensus_queue.cc:260] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:16.124684 30876 tablet_bootstrap.cc:492] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e: No bootstrap required, opened a new log
I20260810 18:50:16.125240 30876 ts_tablet_manager.cc:1397] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.010s
I20260810 18:50:16.127846 30876 raft_consensus.cc:348] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:16.128579 30876 raft_consensus.cc:374] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:16.129043 30876 raft_consensus.cc:729] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f3bcaa02664eb2ad6f3a1db136739e, State: Initialized, Role: FOLLOWER
I20260810 18:50:16.129949 30876 consensus_queue.cc:260] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:16.137320 30608 heartbeater.cc:503] Master 127.28.93.190:43307 was elected leader, sending a full tablet report...
I20260810 18:50:16.137804 30873 ts_tablet_manager.cc:1428] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf: Time spent starting tablet: real 0.020s user 0.005s sys 0.004s
I20260810 18:50:16.140727 30876 ts_tablet_manager.cc:1428] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e: Time spent starting tablet: real 0.015s user 0.005s sys 0.002s
I20260810 18:50:16.141551 30727 heartbeater.cc:503] Master 127.28.93.190:43307 was elected leader, sending a full tablet report...
I20260810 18:50:16.268790 30887 raft_consensus.cc:482] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:16.269270 30887 raft_consensus.cc:504] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:16.271512 30887 leader_election.cc:288] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb9aa4200f654b6786655d8422b3bba5 (127.28.93.131:38381), 34f3bcaa02664eb2ad6f3a1db136739e (127.28.93.130:33523)
I20260810 18:50:16.284885 30807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab69f775f9e452ba812bc35e127fe1e" candidate_uuid: "486704a2d6f04d019f30ce822842aedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb9aa4200f654b6786655d8422b3bba5" is_pre_election: true
I20260810 18:50:16.285856 30807 raft_consensus.cc:2454] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 486704a2d6f04d019f30ce822842aedf in term 0.
I20260810 18:50:16.286154 30689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab69f775f9e452ba812bc35e127fe1e" candidate_uuid: "486704a2d6f04d019f30ce822842aedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" is_pre_election: true
I20260810 18:50:16.286772 30689 raft_consensus.cc:2454] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 486704a2d6f04d019f30ce822842aedf in term 0.
I20260810 18:50:16.287150 30528 leader_election.cc:302] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 486704a2d6f04d019f30ce822842aedf, fb9aa4200f654b6786655d8422b3bba5; no voters:
I20260810 18:50:16.287881 30887 raft_consensus.cc:2781] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:16.288134 30887 raft_consensus.cc:482] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:16.288393 30887 raft_consensus.cc:3037] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:16.293018 30887 raft_consensus.cc:504] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:16.294394 30887 leader_election.cc:288] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [CANDIDATE]: Term 1 election: Requested vote from peers fb9aa4200f654b6786655d8422b3bba5 (127.28.93.131:38381), 34f3bcaa02664eb2ad6f3a1db136739e (127.28.93.130:33523)
I20260810 18:50:16.295195 30807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab69f775f9e452ba812bc35e127fe1e" candidate_uuid: "486704a2d6f04d019f30ce822842aedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb9aa4200f654b6786655d8422b3bba5"
I20260810 18:50:16.295405 30689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab69f775f9e452ba812bc35e127fe1e" candidate_uuid: "486704a2d6f04d019f30ce822842aedf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34f3bcaa02664eb2ad6f3a1db136739e"
I20260810 18:50:16.295619 30807 raft_consensus.cc:3037] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:16.295969 30689 raft_consensus.cc:3037] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:16.302124 30689 raft_consensus.cc:2454] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 486704a2d6f04d019f30ce822842aedf in term 1.
I20260810 18:50:16.302124 30807 raft_consensus.cc:2454] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 486704a2d6f04d019f30ce822842aedf in term 1.
I20260810 18:50:16.303149 30528 leader_election.cc:302] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34f3bcaa02664eb2ad6f3a1db136739e, 486704a2d6f04d019f30ce822842aedf; no voters:
I20260810 18:50:16.303746 30887 raft_consensus.cc:2781] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:16.304525 30887 raft_consensus.cc:686] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 1 LEADER]: Becoming Leader. State: Replica: 486704a2d6f04d019f30ce822842aedf, State: Running, Role: LEADER
I20260810 18:50:16.305145 30887 consensus_queue.cc:237] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:16.310853 30420 catalog_manager.cc:5495] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf reported cstate change: term changed from 0 to 1, leader changed from <none> to 486704a2d6f04d019f30ce822842aedf (127.28.93.129). New cstate: current_term: 1 leader_uuid: "486704a2d6f04d019f30ce822842aedf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:16.772097 30887 consensus_queue.cc:1035] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:16.782374 30906 consensus_queue.cc:1035] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:18.369599 29046 tablet_replica.cc:330] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf: stopping tablet replica
I20260810 18:50:18.370182 29046 raft_consensus.cc:2229] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:18.370908 29046 raft_consensus.cc:2258] T bab69f775f9e452ba812bc35e127fe1e P 486704a2d6f04d019f30ce822842aedf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:19.444411 30915 raft_consensus.cc:482] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 486704a2d6f04d019f30ce822842aedf)
I20260810 18:50:19.444916 30915 raft_consensus.cc:504] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:19.446751 30915 leader_election.cc:288] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 486704a2d6f04d019f30ce822842aedf (127.28.93.129:46741), fb9aa4200f654b6786655d8422b3bba5 (127.28.93.131:38381)
I20260810 18:50:19.461746 30807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab69f775f9e452ba812bc35e127fe1e" candidate_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fb9aa4200f654b6786655d8422b3bba5" is_pre_election: true
I20260810 18:50:19.462313 30575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab69f775f9e452ba812bc35e127fe1e" candidate_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "486704a2d6f04d019f30ce822842aedf" is_pre_election: true
W20260810 18:50:19.463505 30636 leader_election.cc:341] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 486704a2d6f04d019f30ce822842aedf (127.28.93.129:46741): Illegal state: cannot vote while shut down
I20260810 18:50:19.463900 30636 leader_election.cc:302] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34f3bcaa02664eb2ad6f3a1db136739e; no voters: 486704a2d6f04d019f30ce822842aedf, fb9aa4200f654b6786655d8422b3bba5
I20260810 18:50:19.464483 30915 raft_consensus.cc:2726] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:50:19.838022 30922 raft_consensus.cc:482] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 486704a2d6f04d019f30ce822842aedf)
I20260810 18:50:19.838502 30922 raft_consensus.cc:504] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:19.840550 30922 leader_election.cc:288] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 486704a2d6f04d019f30ce822842aedf (127.28.93.129:46741), 34f3bcaa02664eb2ad6f3a1db136739e (127.28.93.130:33523)
I20260810 18:50:19.853638 30575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab69f775f9e452ba812bc35e127fe1e" candidate_uuid: "fb9aa4200f654b6786655d8422b3bba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "486704a2d6f04d019f30ce822842aedf" is_pre_election: true
W20260810 18:50:19.855149 30746 leader_election.cc:341] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 486704a2d6f04d019f30ce822842aedf (127.28.93.129:46741): Illegal state: cannot vote while shut down
I20260810 18:50:19.855118 30689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab69f775f9e452ba812bc35e127fe1e" candidate_uuid: "fb9aa4200f654b6786655d8422b3bba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" is_pre_election: true
I20260810 18:50:19.855818 30689 raft_consensus.cc:2454] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb9aa4200f654b6786655d8422b3bba5 in term 1.
I20260810 18:50:19.856719 30750 leader_election.cc:302] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34f3bcaa02664eb2ad6f3a1db136739e, fb9aa4200f654b6786655d8422b3bba5; no voters: 486704a2d6f04d019f30ce822842aedf
I20260810 18:50:19.857359 30922 raft_consensus.cc:2781] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:50:19.857674 30922 raft_consensus.cc:482] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 486704a2d6f04d019f30ce822842aedf)
I20260810 18:50:19.857901 30922 raft_consensus.cc:3037] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:19.862151 30922 raft_consensus.cc:504] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:19.863469 30922 leader_election.cc:288] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [CANDIDATE]: Term 2 election: Requested vote from peers 486704a2d6f04d019f30ce822842aedf (127.28.93.129:46741), 34f3bcaa02664eb2ad6f3a1db136739e (127.28.93.130:33523)
I20260810 18:50:19.864122 30575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab69f775f9e452ba812bc35e127fe1e" candidate_uuid: "fb9aa4200f654b6786655d8422b3bba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "486704a2d6f04d019f30ce822842aedf"
I20260810 18:50:19.864392 30689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab69f775f9e452ba812bc35e127fe1e" candidate_uuid: "fb9aa4200f654b6786655d8422b3bba5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "34f3bcaa02664eb2ad6f3a1db136739e"
I20260810 18:50:19.864835 30689 raft_consensus.cc:3037] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:50:19.865085 30746 leader_election.cc:341] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 486704a2d6f04d019f30ce822842aedf (127.28.93.129:46741): Illegal state: cannot vote while shut down
I20260810 18:50:19.869132 30689 raft_consensus.cc:2454] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb9aa4200f654b6786655d8422b3bba5 in term 2.
I20260810 18:50:19.870005 30750 leader_election.cc:302] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 34f3bcaa02664eb2ad6f3a1db136739e, fb9aa4200f654b6786655d8422b3bba5; no voters: 486704a2d6f04d019f30ce822842aedf
I20260810 18:50:19.870599 30922 raft_consensus.cc:2781] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:19.871601 30922 raft_consensus.cc:686] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 2 LEADER]: Becoming Leader. State: Replica: fb9aa4200f654b6786655d8422b3bba5, State: Running, Role: LEADER
I20260810 18:50:19.872249 30922 consensus_queue.cc:237] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486704a2d6f04d019f30ce822842aedf" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 46741 } } peers { permanent_uuid: "fb9aa4200f654b6786655d8422b3bba5" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 38381 } } peers { permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 } }
I20260810 18:50:20.356544 30746 consensus_queue.cc:579] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [LEADER]: The tablet replica hosted on peer 486704a2d6f04d019f30ce822842aedf has failed
W20260810 18:50:20.357045 30746 consensus_peers.cc:487] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 -> Peer 486704a2d6f04d019f30ce822842aedf (127.28.93.129:46741): Couldn't send request to peer 486704a2d6f04d019f30ce822842aedf. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:20.421582 30689 raft_consensus.cc:1261] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 2 FOLLOWER]: Refusing update from remote peer fb9aa4200f654b6786655d8422b3bba5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:20.422770 30922 consensus_queue.cc:1035] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f3bcaa02664eb2ad6f3a1db136739e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 33523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:21.405797 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
W20260810 18:50:21.415557 30746 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260810 18:50:21.424816 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:21.441679 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:21.448870 29046 tablet_server.cc:178] TabletServer@127.28.93.130:0 shutting down...
I20260810 18:50:21.462700 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:21.463212 29046 tablet_replica.cc:330] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e: stopping tablet replica
I20260810 18:50:21.463660 29046 raft_consensus.cc:2229] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:21.464076 29046 raft_consensus.cc:2258] T bab69f775f9e452ba812bc35e127fe1e P 34f3bcaa02664eb2ad6f3a1db136739e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:21.481400 29046 tablet_server.cc:194] TabletServer@127.28.93.130:0 shutdown complete.
I20260810 18:50:21.488445 29046 tablet_server.cc:178] TabletServer@127.28.93.131:0 shutting down...
I20260810 18:50:21.501960 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:21.502501 29046 tablet_replica.cc:330] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5: stopping tablet replica
I20260810 18:50:21.502941 29046 raft_consensus.cc:2229] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 2 LEADER]: Raft consensus shutting down.
I20260810 18:50:21.503631 29046 raft_consensus.cc:2258] T bab69f775f9e452ba812bc35e127fe1e P fb9aa4200f654b6786655d8422b3bba5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:21.520701 29046 tablet_server.cc:194] TabletServer@127.28.93.131:0 shutdown complete.
I20260810 18:50:21.528586 29046 master.cc:537] Master@127.28.93.190:43307 shutting down...
I20260810 18:50:21.543107 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:21.543572 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:21.543901 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8e0857e664c14bcb8abd8ef3f5359d23: stopping tablet replica
I20260810 18:50:21.560065 29046 master.cc:552] Master@127.28.93.190:43307 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (6138 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20260810 18:50:21.581876 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:41947
I20260810 18:50:21.582834 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:21.588289 30952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:21.588768 30950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:21.590022 30956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:21.591105 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:21.591725 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:21.591892 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:21.592015 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387821592005 us; error 0 us; skew 500 ppm
I20260810 18:50:21.592449 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:21.598613 29046 webserver.cc:460] Webserver started at http://127.28.93.190:34689/ using document root <none> and password file <none>
I20260810 18:50:21.599028 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:21.599170 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:21.599359 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:21.600449 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "3927c0162fd648e29eb7cd8f1d9d0c89"
format_stamp: "Formatted at 2026-08-10 18:50:21 on dist-test-slave-nj21"
I20260810 18:50:21.604487 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:50:21.607479 30963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:21.608208 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:21.608461 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "3927c0162fd648e29eb7cd8f1d9d0c89"
format_stamp: "Formatted at 2026-08-10 18:50:21 on dist-test-slave-nj21"
I20260810 18:50:21.608711 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:21.621428 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:21.622303 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:21.655309 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:41947
I20260810 18:50:21.655395 31062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:41947 every 8 connection(s)
I20260810 18:50:21.658818 31063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:21.667554 31063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89: Bootstrap starting.
I20260810 18:50:21.671320 31063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:21.674841 31063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89: No bootstrap required, opened a new log
I20260810 18:50:21.676990 31063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927c0162fd648e29eb7cd8f1d9d0c89" member_type: VOTER }
I20260810 18:50:21.677471 31063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:21.677762 31063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3927c0162fd648e29eb7cd8f1d9d0c89, State: Initialized, Role: FOLLOWER
I20260810 18:50:21.678462 31063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927c0162fd648e29eb7cd8f1d9d0c89" member_type: VOTER }
I20260810 18:50:21.679041 31063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:21.679327 31063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:21.679664 31063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:21.683499 31063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927c0162fd648e29eb7cd8f1d9d0c89" member_type: VOTER }
I20260810 18:50:21.683975 31063 leader_election.cc:302] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3927c0162fd648e29eb7cd8f1d9d0c89; no voters:
I20260810 18:50:21.684954 31063 leader_election.cc:288] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:21.685220 31067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:21.686389 31067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 1 LEADER]: Becoming Leader. State: Replica: 3927c0162fd648e29eb7cd8f1d9d0c89, State: Running, Role: LEADER
I20260810 18:50:21.686962 31067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927c0162fd648e29eb7cd8f1d9d0c89" member_type: VOTER }
I20260810 18:50:21.687566 31063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:21.689483 31069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3927c0162fd648e29eb7cd8f1d9d0c89. Latest consensus state: current_term: 1 leader_uuid: "3927c0162fd648e29eb7cd8f1d9d0c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927c0162fd648e29eb7cd8f1d9d0c89" member_type: VOTER } }
I20260810 18:50:21.689528 31068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3927c0162fd648e29eb7cd8f1d9d0c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3927c0162fd648e29eb7cd8f1d9d0c89" member_type: VOTER } }
I20260810 18:50:21.690202 31069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:21.690339 31068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:21.692528 31082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:21.696893 31082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:21.701618 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:21.704368 31082 catalog_manager.cc:1347] Generated new cluster ID: 5adea484b3514dab8e754561293c9bbf
I20260810 18:50:21.704609 31082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:21.723204 31082 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:21.724807 31082 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:21.739130 31082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89: Generated new TSK 0
I20260810 18:50:21.739749 31082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:21.767575 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:21.773123 31099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:21.773965 31100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:21.775688 31103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:21.776114 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:21.776932 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:21.777112 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:21.777256 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387821777241 us; error 0 us; skew 500 ppm
I20260810 18:50:21.777733 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:21.779968 29046 webserver.cc:460] Webserver started at http://127.28.93.129:41669/ using document root <none> and password file <none>
I20260810 18:50:21.780418 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:21.780597 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:21.780835 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:21.781863 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "ce706a23c695428f98e107947474650d"
format_stamp: "Formatted at 2026-08-10 18:50:21 on dist-test-slave-nj21"
I20260810 18:50:21.786124 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:50:21.789330 31113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:21.790011 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:50:21.790263 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "ce706a23c695428f98e107947474650d"
format_stamp: "Formatted at 2026-08-10 18:50:21 on dist-test-slave-nj21"
I20260810 18:50:21.790515 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:21.802220 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:21.803187 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:21.804487 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:21.806535 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:21.806699 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:21.806913 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:21.807053 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:21.841969 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:41839
I20260810 18:50:21.842063 31215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:41839 every 8 connection(s)
I20260810 18:50:21.856199 31216 heartbeater.cc:347] Connected to a master server at 127.28.93.190:41947
I20260810 18:50:21.856523 31216 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:21.857106 31216 heartbeater.cc:511] Master 127.28.93.190:41947 requested a full tablet report, sending...
I20260810 18:50:21.858764 30992 ts_manager.cc:194] Registered new tserver with Master: ce706a23c695428f98e107947474650d (127.28.93.129:41839)
I20260810 18:50:21.858896 29046 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.01401308s
I20260810 18:50:21.861439 30992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46184
I20260810 18:50:21.879555 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:21.892536 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:21.907341 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:21.914080 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:50:21.916612 29046 tablet_server.cc:93] deduplicated master addresses: 127.28.93.190:41947
W20260810 18:50:21.919364 31231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:21.920979 31233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:21.921142 31237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:21.921680 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:21.922468 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:21.922643 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:21.922786 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387821922775 us; error 0 us; skew 500 ppm
I20260810 18:50:21.923280 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:21.925400 29046 webserver.cc:460] Webserver started at http://127.28.93.129:41669/ using document root <none> and password file <none>
I20260810 18:50:21.925837 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:21.925995 29046 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260810 18:50:21.929266 29046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:50:21.931581 31242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:21.932283 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 18:50:21.932539 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "ce706a23c695428f98e107947474650d"
format_stamp: "Formatted at 2026-08-10 18:50:21 on dist-test-slave-nj21"
I20260810 18:50:21.932799 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:21.951309 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:21.952221 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:21.953517 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:21.955570 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:21.955762 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:21.955973 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:21.956116 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:21.992316 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:41839
I20260810 18:50:21.992465 31341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:41839 every 8 connection(s)
I20260810 18:50:21.998260 29046 tablet_server.cc:178] TabletServer@127.28.93.129:41839 shutting down...
I20260810 18:50:22.013365 31342 heartbeater.cc:347] Connected to a master server at 127.28.93.190:41947
I20260810 18:50:22.013832 31342 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:22.014739 31342 heartbeater.cc:511] Master 127.28.93.190:41947 requested a full tablet report, sending...
I20260810 18:50:22.017293 30992 ts_manager.cc:194] Re-registered known tserver with Master: ce706a23c695428f98e107947474650d (127.28.93.129:41839)
I20260810 18:50:22.018880 30992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46198
I20260810 18:50:22.021440 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:22.036177 29046 tablet_server.cc:194] TabletServer@127.28.93.129:41839 shutdown complete.
I20260810 18:50:22.042414 29046 master.cc:537] Master@127.28.93.190:41947 shutting down...
I20260810 18:50:22.055130 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:22.055567 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:22.056007 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3927c0162fd648e29eb7cd8f1d9d0c89: stopping tablet replica
I20260810 18:50:22.064038 29046 master.cc:552] Master@127.28.93.190:41947 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (501 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20260810 18:50:22.083441 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:38651,127.28.93.189:43453,127.28.93.188:36699
I20260810 18:50:22.084612 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:22.088924 31353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:22.090081 31354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:22.091535 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:22.092201 31357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:22.092880 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:22.093050 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:22.093194 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387822093179 us; error 0 us; skew 500 ppm
I20260810 18:50:22.093755 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:22.095875 29046 webserver.cc:460] Webserver started at http://127.28.93.190:33385/ using document root <none> and password file <none>
I20260810 18:50:22.096290 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:22.096459 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:22.096691 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:22.097700 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "f7b3db47356949cc9049320effc90535"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:22.101877 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 18:50:22.105031 31364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:22.105712 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:50:22.105981 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "f7b3db47356949cc9049320effc90535"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:22.106248 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:22.124087 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:22.125056 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:22.158389 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:38651
I20260810 18:50:22.158493 31445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:38651 every 8 connection(s)
I20260810 18:50:22.161944 31446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:22.162101 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:22.167197 31449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:22.168597 31446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } has no permanent_uuid. Determining permanent_uuid...
W20260810 18:50:22.169322 31450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:22.170298 31452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:22.172065 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:22.172885 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:22.173094 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:22.173303 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387822173285 us; error 0 us; skew 500 ppm
I20260810 18:50:22.173795 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:22.176371 29046 webserver.cc:460] Webserver started at http://127.28.93.189:43141/ using document root <none> and password file <none>
I20260810 18:50:22.176903 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:22.177068 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:22.177304 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:22.178279 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-1-root/instance:
uuid: "79547306558a49778d11f86a5276bb76"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:22.183790 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 18:50:22.183844 31446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:50:22.187649 31467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:22.188431 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
W20260810 18:50:22.188709 31446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.93.189:43453: Network error: Client connection negotiation failed: client connection to 127.28.93.189:43453: connect: Connection refused (error 111)
I20260810 18:50:22.188714 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-1-root
uuid: "79547306558a49778d11f86a5276bb76"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:22.189141 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:22.224172 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:22.225134 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:22.227262 31446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } attempt: 1
W20260810 18:50:22.230865 31446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.93.189:43453: Network error: Client connection negotiation failed: client connection to 127.28.93.189:43453: connect: Connection refused (error 111)
I20260810 18:50:22.258136 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.189:43453
I20260810 18:50:22.258217 31543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.189:43453 every 8 connection(s)
I20260810 18:50:22.261543 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:50:22.261730 31545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 18:50:22.266369 31548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:22.267485 31552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:22.269820 31545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:50:22.270120 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:22.270498 31554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:22.271263 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260810 18:50:22.271314 31446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } attempt: 2
W20260810 18:50:22.271617 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:22.271870 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387822271851 us; error 0 us; skew 500 ppm
I20260810 18:50:22.272557 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:22.276121 29046 webserver.cc:460] Webserver started at http://127.28.93.188:45095/ using document root <none> and password file <none>
I20260810 18:50:22.276695 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:22.276927 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:22.277252 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:22.278648 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-2-root/instance:
uuid: "bc51947e5d9c444e85b6c335c9cf79d8"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:22.284754 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 18:50:22.285300 31545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:50:22.288159 31446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:50:22.290604 31562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260810 18:50:22.291973 31446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.93.188:36699: Network error: Client connection negotiation failed: client connection to 127.28.93.188:36699: connect: Connection refused (error 111)
I20260810 18:50:22.292796 29046 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260810 18:50:22.293140 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-2-root
uuid: "bc51947e5d9c444e85b6c335c9cf79d8"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:22.293481 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:22.297432 31545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } has no permanent_uuid. Determining permanent_uuid...
W20260810 18:50:22.302181 31545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.93.188:36699: Network error: Client connection negotiation failed: client connection to 127.28.93.188:36699: connect: Connection refused (error 111)
I20260810 18:50:22.309998 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:22.310896 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:22.327493 31446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } attempt: 1
W20260810 18:50:22.332835 31446 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.93.188:36699: Network error: Client connection negotiation failed: client connection to 127.28.93.188:36699: connect: Connection refused (error 111)
I20260810 18:50:22.335774 31545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } attempt: 1
W20260810 18:50:22.339686 31545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.93.188:36699: Network error: Client connection negotiation failed: client connection to 127.28.93.188:36699: connect: Connection refused (error 111)
I20260810 18:50:22.347901 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.188:36699
I20260810 18:50:22.347990 31651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.188:36699 every 8 connection(s)
I20260810 18:50:22.350726 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:22.351706 31652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:22.356889 31652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:50:22.368499 31652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:50:22.378144 31545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } attempt: 2
I20260810 18:50:22.378798 31652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:50:22.399966 31545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: Bootstrap starting.
I20260810 18:50:22.400841 31652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: Bootstrap starting.
I20260810 18:50:22.403946 31545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:22.404665 31652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:22.408397 31545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: No bootstrap required, opened a new log
I20260810 18:50:22.408448 31446 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } attempt: 2
I20260810 18:50:22.409011 31652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: No bootstrap required, opened a new log
I20260810 18:50:22.410748 31545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:22.410949 31652 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:22.411500 31545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:22.411582 31652 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:22.411958 31652 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc51947e5d9c444e85b6c335c9cf79d8, State: Initialized, Role: FOLLOWER
I20260810 18:50:22.411864 31545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79547306558a49778d11f86a5276bb76, State: Initialized, Role: FOLLOWER
I20260810 18:50:22.412657 31652 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:22.412781 31545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:22.414759 31661 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:22.415688 31661 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:22.416283 31652 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:22.416862 31662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:22.417692 31662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:22.418349 31545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:22.423403 31662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:22.423828 31662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:22.425793 31662 leader_election.cc:288] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7b3db47356949cc9049320effc90535 (127.28.93.190:38651), bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699)
I20260810 18:50:22.428304 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79547306558a49778d11f86a5276bb76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" is_pre_election: true
I20260810 18:50:22.428947 31604 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79547306558a49778d11f86a5276bb76 in term 0.
I20260810 18:50:22.429899 31446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: Bootstrap starting.
I20260810 18:50:22.430274 31469 leader_election.cc:302] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79547306558a49778d11f86a5276bb76, bc51947e5d9c444e85b6c335c9cf79d8; no voters:
I20260810 18:50:22.431192 31662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:22.431627 31662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:22.432076 31662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:22.435351 31446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:22.441521 31662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:22.447748 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79547306558a49778d11f86a5276bb76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc51947e5d9c444e85b6c335c9cf79d8"
I20260810 18:50:22.448310 31604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:22.456640 31662 leader_election.cc:288] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [CANDIDATE]: Term 1 election: Requested vote from peers f7b3db47356949cc9049320effc90535 (127.28.93.190:38651), bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699)
I20260810 18:50:22.457957 31604 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79547306558a49778d11f86a5276bb76 in term 1.
I20260810 18:50:22.459394 31469 leader_election.cc:302] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79547306558a49778d11f86a5276bb76, bc51947e5d9c444e85b6c335c9cf79d8; no voters:
I20260810 18:50:22.460750 31662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:22.462141 31446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: No bootstrap required, opened a new log
I20260810 18:50:22.473148 31446 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:22.475270 31446 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:22.475963 31446 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b3db47356949cc9049320effc90535, State: Initialized, Role: FOLLOWER
I20260810 18:50:22.477141 31662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 1 LEADER]: Becoming Leader. State: Replica: 79547306558a49778d11f86a5276bb76, State: Running, Role: LEADER
I20260810 18:50:22.478065 31662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:22.479130 31446 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:22.490303 31405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79547306558a49778d11f86a5276bb76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b3db47356949cc9049320effc90535"
I20260810 18:50:22.491008 31405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:22.495819 31406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79547306558a49778d11f86a5276bb76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b3db47356949cc9049320effc90535" is_pre_election: true
I20260810 18:50:22.499797 31405 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79547306558a49778d11f86a5276bb76 in term 1.
I20260810 18:50:22.501364 31446 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:22.504897 31690 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:22.505719 31690 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:22.509320 31680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79547306558a49778d11f86a5276bb76. Latest consensus state: current_term: 1 leader_uuid: "79547306558a49778d11f86a5276bb76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:22.510170 31680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:22.515555 31694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:22.525195 31694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 18:50:22.540633 31703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:50:22.541003 31703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:50:22.542296 31604 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 1 FOLLOWER]: Refusing update from remote peer 79547306558a49778d11f86a5276bb76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:50:22.544060 31405 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 1 FOLLOWER]: Refusing update from remote peer 79547306558a49778d11f86a5276bb76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:50:22.545761 31680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:22.547165 31680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:22.553073 31661 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79547306558a49778d11f86a5276bb76. Latest consensus state: current_term: 1 leader_uuid: "79547306558a49778d11f86a5276bb76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:22.553784 31661 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:22.556978 31706 mvcc.cc:205] Tried to move back new op lower bound from 7317044521098563584 to 7317044520881868800. Current Snapshot: MvccSnapshot[applied={T|T < 7317044521098563584}]
I20260810 18:50:22.569357 31680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79547306558a49778d11f86a5276bb76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:22.570127 31680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:22.571012 31694 catalog_manager.cc:1347] Generated new cluster ID: c3bf6b332118469eb0628784c9263293
I20260810 18:50:22.571280 31694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:22.573100 31661 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79547306558a49778d11f86a5276bb76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:22.574494 31661 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:22.576002 31690 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79547306558a49778d11f86a5276bb76. Latest consensus state: current_term: 1 leader_uuid: "79547306558a49778d11f86a5276bb76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:22.576711 31690 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:22.577488 31662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79547306558a49778d11f86a5276bb76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:22.578169 31662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:22.582104 31690 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79547306558a49778d11f86a5276bb76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:22.582760 31690 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:22.588195 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20260810 18:50:22.588483 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20260810 18:50:22.590372 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:50:22.593639 31726 catalog_manager.cc:1259] Loaded cluster ID: c3bf6b332118469eb0628784c9263293
I20260810 18:50:22.593899 31726 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: loading cluster ID for follower catalog manager: success
W20260810 18:50:22.597805 31729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:22.597949 31726 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260810 18:50:22.598342 31726 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20260810 18:50:22.599191 31730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:22.601764 31732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:22.601825 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:22.603020 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:22.603220 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:22.603384 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387822603367 us; error 0 us; skew 500 ppm
I20260810 18:50:22.603909 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:22.606707 29046 webserver.cc:460] Webserver started at http://127.28.93.129:43079/ using document root <none> and password file <none>
I20260810 18:50:22.607158 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:22.607331 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:22.607571 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:22.608626 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "770b39bfac164d40a82408baae0ab88e"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:22.613031 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:50:22.616210 31741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:22.617017 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 18:50:22.617349 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "770b39bfac164d40a82408baae0ab88e"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:22.617682 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:22.636869 31694 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:22.639003 31694 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:22.644059 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:22.645334 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:22.646983 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:22.649799 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:22.650041 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:22.650282 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:22.650460 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:22.664798 31694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: Generated new TSK 0
I20260810 18:50:22.665541 31694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:22.701229 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:43783
I20260810 18:50:22.701330 31850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:43783 every 8 connection(s)
I20260810 18:50:22.711000 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:22.720906 31870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:22.731896 31871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:22.742228 31855 heartbeater.cc:347] Connected to a master server at 127.28.93.188:36699
I20260810 18:50:22.742985 31855 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:22.741034 31861 heartbeater.cc:347] Connected to a master server at 127.28.93.190:38651
I20260810 18:50:22.745223 31861 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:22.745899 31855 heartbeater.cc:511] Master 127.28.93.188:36699 requested a full tablet report, sending...
I20260810 18:50:22.746899 31861 heartbeater.cc:511] Master 127.28.93.190:38651 requested a full tablet report, sending...
I20260810 18:50:22.753333 31589 ts_manager.cc:194] Registered new tserver with Master: 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:22.755352 31392 ts_manager.cc:194] Registered new tserver with Master: 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:22.761189 31859 heartbeater.cc:347] Connected to a master server at 127.28.93.189:43453
I20260810 18:50:22.761714 31859 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:22.762847 31859 heartbeater.cc:511] Master 127.28.93.189:43453 requested a full tablet report, sending...
I20260810 18:50:22.767760 31494 ts_manager.cc:194] Registered new tserver with Master: 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:22.781020 31494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50284
I20260810 18:50:22.791157 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:22.792135 31876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:22.793080 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:22.793286 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:22.793447 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387822793433 us; error 0 us; skew 500 ppm
I20260810 18:50:22.793921 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:22.796375 29046 webserver.cc:460] Webserver started at http://127.28.93.130:46783/ using document root <none> and password file <none>
I20260810 18:50:22.796820 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:22.796998 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:22.797240 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:22.798251 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-1-root/instance:
uuid: "f58326f8fbd44e1984c2ce30da077d2e"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:22.802580 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:50:22.805745 31883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:22.806661 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:50:22.806936 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-1-root
uuid: "f58326f8fbd44e1984c2ce30da077d2e"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:22.807192 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:22.826663 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:22.829499 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:22.832598 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:22.838370 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:22.838568 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:22.838792 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:22.838970 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:22.889155 31859 heartbeater.cc:503] Master 127.28.93.189:43453 was elected leader, sending a full tablet report...
I20260810 18:50:23.090085 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.130:45437
I20260810 18:50:23.093334 31976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.130:45437 every 8 connection(s)
I20260810 18:50:23.127807 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:23.148723 31990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:23.154701 31977 heartbeater.cc:347] Connected to a master server at 127.28.93.188:36699
I20260810 18:50:23.155335 31977 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:23.157392 31977 heartbeater.cc:511] Master 127.28.93.188:36699 requested a full tablet report, sending...
W20260810 18:50:23.166193 31991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:23.167213 31589 ts_manager.cc:194] Registered new tserver with Master: f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:23.170341 31978 heartbeater.cc:347] Connected to a master server at 127.28.93.189:43453
I20260810 18:50:23.170747 31978 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:23.171521 31978 heartbeater.cc:511] Master 127.28.93.189:43453 requested a full tablet report, sending...
I20260810 18:50:23.174706 31494 ts_manager.cc:194] Registered new tserver with Master: f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:23.178903 31494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50300
W20260810 18:50:23.183445 31998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:23.184403 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:23.185201 31982 heartbeater.cc:347] Connected to a master server at 127.28.93.190:38651
I20260810 18:50:23.185571 31982 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:23.186219 31982 heartbeater.cc:511] Master 127.28.93.190:38651 requested a full tablet report, sending...
I20260810 18:50:23.186614 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:23.186883 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:23.187088 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387823187072 us; error 0 us; skew 500 ppm
I20260810 18:50:23.187603 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:23.188517 31390 ts_manager.cc:194] Registered new tserver with Master: f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:23.190208 29046 webserver.cc:460] Webserver started at http://127.28.93.131:33327/ using document root <none> and password file <none>
I20260810 18:50:23.190696 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:23.190891 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:23.191126 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:23.192188 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-2-root/instance:
uuid: "8314391cfbd645b4b780d02a43b4661d"
format_stamp: "Formatted at 2026-08-10 18:50:23 on dist-test-slave-nj21"
I20260810 18:50:23.196393 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 18:50:23.199429 32007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:23.200137 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 18:50:23.200398 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-2-root
uuid: "8314391cfbd645b4b780d02a43b4661d"
format_stamp: "Formatted at 2026-08-10 18:50:23 on dist-test-slave-nj21"
I20260810 18:50:23.200645 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:23.216598 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:23.218025 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:23.221405 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:23.224607 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:23.224810 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:23.225020 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:23.225165 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:23.285135 31978 heartbeater.cc:503] Master 127.28.93.189:43453 was elected leader, sending a full tablet report...
I20260810 18:50:23.481256 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.131:37737
I20260810 18:50:23.481637 32113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.131:37737 every 8 connection(s)
I20260810 18:50:23.557596 32115 heartbeater.cc:347] Connected to a master server at 127.28.93.188:36699
I20260810 18:50:23.558329 32115 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:23.561105 32115 heartbeater.cc:511] Master 127.28.93.188:36699 requested a full tablet report, sending...
I20260810 18:50:23.555039 31703 catalog_manager.cc:1259] Loaded cluster ID: c3bf6b332118469eb0628784c9263293
I20260810 18:50:23.563901 32116 heartbeater.cc:347] Connected to a master server at 127.28.93.189:43453
I20260810 18:50:23.564870 31703 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: loading cluster ID for follower catalog manager: success
I20260810 18:50:23.565193 32116 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:23.565567 31589 ts_manager.cc:194] Registered new tserver with Master: 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:23.566759 32116 heartbeater.cc:511] Master 127.28.93.189:43453 requested a full tablet report, sending...
I20260810 18:50:23.571590 31494 ts_manager.cc:194] Registered new tserver with Master: 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:23.576303 31494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50304
I20260810 18:50:23.577997 31703 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: acquiring CA information for follower catalog manager: success
I20260810 18:50:23.579780 32120 heartbeater.cc:347] Connected to a master server at 127.28.93.190:38651
I20260810 18:50:23.580386 32120 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:23.581773 32120 heartbeater.cc:511] Master 127.28.93.190:38651 requested a full tablet report, sending...
I20260810 18:50:23.586269 31703 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:50:23.589068 31390 ts_manager.cc:194] Registered new tserver with Master: 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:23.590252 29046 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.025414477s
I20260810 18:50:23.611183 31726 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: acquiring CA information for follower catalog manager: success
I20260810 18:50:23.619300 31726 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:50:23.682680 32116 heartbeater.cc:503] Master 127.28.93.189:43453 was elected leader, sending a full tablet report...
I20260810 18:50:23.684535 31494 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20260810 18:50:23.688047 31494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:50:23.869287 31924 tablet_service.cc:1467] Processing CreateTablet for tablet 0f88e634539c49fd95f67aaedf60a5b5 (DEFAULT_TABLE table=test-table [id=c384a89a9c1b42f8ae3b3f1d6815aba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:23.873863 31924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f88e634539c49fd95f67aaedf60a5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:23.871302 31923 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0d266df91e4d60b5289e0683068f80 (DEFAULT_TABLE table=test-table [id=c384a89a9c1b42f8ae3b3f1d6815aba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:23.879742 31923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0d266df91e4d60b5289e0683068f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:23.876291 31925 tablet_service.cc:1467] Processing CreateTablet for tablet 618c53eed4a94a9eab8d7c30da86b31f (DEFAULT_TABLE table=test-table [id=c384a89a9c1b42f8ae3b3f1d6815aba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:23.883615 31925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618c53eed4a94a9eab8d7c30da86b31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:23.891757 32050 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0d266df91e4d60b5289e0683068f80 (DEFAULT_TABLE table=test-table [id=c384a89a9c1b42f8ae3b3f1d6815aba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:23.890596 32052 tablet_service.cc:1467] Processing CreateTablet for tablet 618c53eed4a94a9eab8d7c30da86b31f (DEFAULT_TABLE table=test-table [id=c384a89a9c1b42f8ae3b3f1d6815aba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:23.890697 32051 tablet_service.cc:1467] Processing CreateTablet for tablet 0f88e634539c49fd95f67aaedf60a5b5 (DEFAULT_TABLE table=test-table [id=c384a89a9c1b42f8ae3b3f1d6815aba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:23.898033 32050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0d266df91e4d60b5289e0683068f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:23.912915 32052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618c53eed4a94a9eab8d7c30da86b31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:23.920135 32051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f88e634539c49fd95f67aaedf60a5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:23.932780 31790 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0d266df91e4d60b5289e0683068f80 (DEFAULT_TABLE table=test-table [id=c384a89a9c1b42f8ae3b3f1d6815aba9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:23.934170 31791 tablet_service.cc:1467] Processing CreateTablet for tablet 0f88e634539c49fd95f67aaedf60a5b5 (DEFAULT_TABLE table=test-table [id=c384a89a9c1b42f8ae3b3f1d6815aba9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:23.939421 31792 tablet_service.cc:1467] Processing CreateTablet for tablet 618c53eed4a94a9eab8d7c30da86b31f (DEFAULT_TABLE table=test-table [id=c384a89a9c1b42f8ae3b3f1d6815aba9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:23.944857 31790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0d266df91e4d60b5289e0683068f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:23.952612 31791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f88e634539c49fd95f67aaedf60a5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:23.957244 31792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618c53eed4a94a9eab8d7c30da86b31f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:24.033049 32165 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap starting.
I20260810 18:50:24.041764 32166 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d: Bootstrap starting.
I20260810 18:50:24.083534 32169 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Bootstrap starting.
I20260810 18:50:24.083855 32166 tablet_bootstrap.cc:654] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:24.090260 32165 tablet_bootstrap.cc:654] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:24.100653 32169 tablet_bootstrap.cc:654] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:24.105545 32166 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d: No bootstrap required, opened a new log
I20260810 18:50:24.106419 32166 ts_tablet_manager.cc:1397] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.017s
I20260810 18:50:24.107712 32165 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: No bootstrap required, opened a new log
I20260810 18:50:24.108342 32165 ts_tablet_manager.cc:1397] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.021s
I20260810 18:50:24.109850 32166 raft_consensus.cc:348] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.110571 32166 raft_consensus.cc:374] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:24.110844 32166 raft_consensus.cc:729] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8314391cfbd645b4b780d02a43b4661d, State: Initialized, Role: FOLLOWER
I20260810 18:50:24.111490 32169 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: No bootstrap required, opened a new log
I20260810 18:50:24.111433 32165 raft_consensus.cc:348] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:24.111642 32166 consensus_queue.cc:260] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.112486 32165 raft_consensus.cc:374] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:24.112802 32165 raft_consensus.cc:729] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Initialized, Role: FOLLOWER
I20260810 18:50:24.113426 32165 consensus_queue.cc:260] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:24.112188 32169 ts_tablet_manager.cc:1397] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.002s
I20260810 18:50:24.116163 32166 ts_tablet_manager.cc:1428] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20260810 18:50:24.118914 32169 raft_consensus.cc:348] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.120244 32169 raft_consensus.cc:374] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:24.120530 32169 raft_consensus.cc:729] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Initialized, Role: FOLLOWER
I20260810 18:50:24.121320 32169 consensus_queue.cc:260] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.127619 32179 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:24.127954 32169 ts_tablet_manager.cc:1428] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260810 18:50:24.128171 32179 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:24.133116 32169 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Bootstrap starting.
I20260810 18:50:24.135835 32179 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737), 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:24.145323 32174 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:24.145859 32174 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.150053 32166 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d: Bootstrap starting.
I20260810 18:50:24.167766 32166 tablet_bootstrap.cc:654] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:24.170531 32174 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:24.187881 32165 ts_tablet_manager.cc:1428] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Time spent starting tablet: real 0.079s user 0.019s sys 0.027s
I20260810 18:50:24.189920 32165 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap starting.
I20260810 18:50:24.199260 32166 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d: No bootstrap required, opened a new log
I20260810 18:50:24.200111 32181 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:24.200616 32181 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.201776 32169 tablet_bootstrap.cc:654] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:24.199786 32166 ts_tablet_manager.cc:1397] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.001s
I20260810 18:50:24.200250 32165 tablet_bootstrap.cc:654] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:24.212636 32181 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:24.217976 32166 raft_consensus.cc:348] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:24.219424 32166 raft_consensus.cc:374] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:24.222469 32166 raft_consensus.cc:729] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8314391cfbd645b4b780d02a43b4661d, State: Initialized, Role: FOLLOWER
I20260810 18:50:24.223311 32166 consensus_queue.cc:260] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:24.227978 32166 ts_tablet_manager.cc:1428] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d: Time spent starting tablet: real 0.025s user 0.005s sys 0.002s
I20260810 18:50:24.233536 32166 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d: Bootstrap starting.
I20260810 18:50:24.235396 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
W20260810 18:50:24.239326 32009 leader_election.cc:341] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:24.242516 32165 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: No bootstrap required, opened a new log
I20260810 18:50:24.243036 32165 ts_tablet_manager.cc:1397] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.004s
I20260810 18:50:24.247047 32166 tablet_bootstrap.cc:654] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:24.252110 32169 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: No bootstrap required, opened a new log
I20260810 18:50:24.254042 32169 ts_tablet_manager.cc:1397] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Time spent bootstrapping tablet: real 0.121s user 0.009s sys 0.013s
I20260810 18:50:24.253155 32165 raft_consensus.cc:348] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.255596 32165 raft_consensus.cc:374] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:24.255589 31940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:24.256446 32165 raft_consensus.cc:729] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Initialized, Role: FOLLOWER
I20260810 18:50:24.259034 32165 consensus_queue.cc:260] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
W20260810 18:50:24.260054 32008 leader_election.cc:341] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:24.260540 32008 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d; no voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:24.263944 32165 ts_tablet_manager.cc:1428] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Time spent starting tablet: real 0.021s user 0.002s sys 0.007s
I20260810 18:50:24.263549 32169 raft_consensus.cc:348] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:24.264523 32174 raft_consensus.cc:2726] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:24.264609 32169 raft_consensus.cc:374] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:24.265456 32169 raft_consensus.cc:729] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Initialized, Role: FOLLOWER
I20260810 18:50:24.266855 32165 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap starting.
I20260810 18:50:24.266847 32169 consensus_queue.cc:260] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:24.270798 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:24.271400 31806 raft_consensus.cc:2454] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 0.
I20260810 18:50:24.272795 32169 ts_tablet_manager.cc:1428] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260810 18:50:24.273837 32169 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Bootstrap starting.
I20260810 18:50:24.294662 32174 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:24.295178 32174 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:24.296485 32176 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:24.297235 32176 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.300335 31940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:24.300724 32165 tablet_bootstrap.cc:654] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:24.298069 32174 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:24.302878 32176 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:24.303114 31940 raft_consensus.cc:2454] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 0.
I20260810 18:50:24.305035 32008 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters:
I20260810 18:50:24.309424 32174 raft_consensus.cc:2781] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:24.310253 32174 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:24.287462 31886 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e; no voters:
I20260810 18:50:24.310735 32174 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:24.313424 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:24.314188 31806 raft_consensus.cc:2454] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 0.
I20260810 18:50:24.315388 32176 raft_consensus.cc:2781] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:24.315800 32176 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:24.316183 32176 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:24.318813 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:24.319973 31806 raft_consensus.cc:2454] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 0.
I20260810 18:50:24.328887 31886 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e; no voters:
I20260810 18:50:24.330119 32179 raft_consensus.cc:2781] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:24.330756 32179 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:24.331451 32179 raft_consensus.cc:3037] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:24.336850 32071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:24.345803 32068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:24.353338 32169 tablet_bootstrap.cc:654] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:24.343869 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
W20260810 18:50:24.378221 31745 leader_election.cc:341] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737): Illegal state: must be running to vote when last-logged opid is not known
W20260810 18:50:24.355706 32008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.28.93.130:45437, user_credentials={real_user=slave}} blocked reactor thread for 50866.5us
I20260810 18:50:24.384410 32174 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
W20260810 18:50:24.388271 31886 leader_election.cc:341] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737): Illegal state: must be running to vote when last-logged opid is not known
I20260810 18:50:24.390264 31940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:24.392482 32068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:24.396958 32071 raft_consensus.cc:2379] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f58326f8fbd44e1984c2ce30da077d2e in current term 1: Already voted for candidate 8314391cfbd645b4b780d02a43b4661d in this term.
I20260810 18:50:24.398466 32166 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d: No bootstrap required, opened a new log
I20260810 18:50:24.399183 32166 ts_tablet_manager.cc:1397] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d: Time spent bootstrapping tablet: real 0.166s user 0.018s sys 0.016s
I20260810 18:50:24.404922 32166 raft_consensus.cc:348] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.406026 32166 raft_consensus.cc:374] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:24.406548 32166 raft_consensus.cc:729] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8314391cfbd645b4b780d02a43b4661d, State: Initialized, Role: FOLLOWER
I20260810 18:50:24.408301 32174 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 1 election: Requested vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:24.409000 32176 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:24.410039 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
I20260810 18:50:24.410704 32166 consensus_queue.cc:260] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.409559 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e"
I20260810 18:50:24.415706 31806 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:24.408381 32179 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.420133 31940 raft_consensus.cc:2379] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 770b39bfac164d40a82408baae0ab88e in current term 1: Already voted for candidate f58326f8fbd44e1984c2ce30da077d2e in this term.
I20260810 18:50:24.417285 32174 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:24.426396 32179 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 election: Requested vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:24.428256 32176 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 election: Requested vote from peers 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737), 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:24.424736 31744 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e; no voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:24.432246 32181 raft_consensus.cc:3037] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:24.424647 32174 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.437078 32166 ts_tablet_manager.cc:1428] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d: Time spent starting tablet: real 0.037s user 0.014s sys 0.003s
I20260810 18:50:24.438373 31938 raft_consensus.cc:2379] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8314391cfbd645b4b780d02a43b4661d in current term 1: Already voted for candidate f58326f8fbd44e1984c2ce30da077d2e in this term.
I20260810 18:50:24.439224 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e"
I20260810 18:50:24.440999 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
I20260810 18:50:24.442808 32069 raft_consensus.cc:2379] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f58326f8fbd44e1984c2ce30da077d2e in current term 1: Already voted for candidate 8314391cfbd645b4b780d02a43b4661d in this term.
I20260810 18:50:24.443624 32174 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:24.443924 32071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
I20260810 18:50:24.444415 32071 raft_consensus.cc:3037] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:24.437240 31805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e"
I20260810 18:50:24.446889 31940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:24.447588 32169 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: No bootstrap required, opened a new log
I20260810 18:50:24.447721 31940 raft_consensus.cc:2379] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8314391cfbd645b4b780d02a43b4661d in current term 1: Already voted for candidate f58326f8fbd44e1984c2ce30da077d2e in this term.
I20260810 18:50:24.448167 32169 ts_tablet_manager.cc:1397] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Time spent bootstrapping tablet: real 0.174s user 0.012s sys 0.015s
I20260810 18:50:24.448670 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:24.452487 32009 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d; no voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:24.452389 31806 raft_consensus.cc:2454] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 1.
I20260810 18:50:24.454455 31886 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58326f8fbd44e1984c2ce30da077d2e; no voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d
I20260810 18:50:24.457367 32169 raft_consensus.cc:348] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.457592 32071 raft_consensus.cc:2454] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 1.
I20260810 18:50:24.458364 32181 raft_consensus.cc:2726] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:24.458843 32169 raft_consensus.cc:374] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:24.459070 32174 raft_consensus.cc:2726] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 18:50:24.459180 32169 raft_consensus.cc:729] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Initialized, Role: FOLLOWER
I20260810 18:50:24.459656 32009 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d; no voters: f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:24.461050 32205 raft_consensus.cc:2781] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:24.460368 32169 consensus_queue.cc:260] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.461920 32176 raft_consensus.cc:2726] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 18:50:24.468819 32165 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: No bootstrap required, opened a new log
I20260810 18:50:24.469352 32165 ts_tablet_manager.cc:1397] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Time spent bootstrapping tablet: real 0.203s user 0.017s sys 0.026s
I20260810 18:50:24.461535 31886 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters:
I20260810 18:50:24.471767 32176 raft_consensus.cc:2781] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:24.473474 32176 raft_consensus.cc:686] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 1 LEADER]: Becoming Leader. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Running, Role: LEADER
I20260810 18:50:24.474318 32176 consensus_queue.cc:237] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.475538 32169 ts_tablet_manager.cc:1428] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20260810 18:50:24.472347 32165 raft_consensus.cc:348] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.477923 31805 raft_consensus.cc:2454] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 1.
I20260810 18:50:24.478822 32165 raft_consensus.cc:374] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:24.479584 32165 raft_consensus.cc:729] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Initialized, Role: FOLLOWER
I20260810 18:50:24.482376 32205 raft_consensus.cc:686] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 1 LEADER]: Becoming Leader. State: Replica: 8314391cfbd645b4b780d02a43b4661d, State: Running, Role: LEADER
I20260810 18:50:24.482903 32165 consensus_queue.cc:260] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.483227 32205 consensus_queue.cc:237] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:24.484872 32179 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:24.485584 32165 ts_tablet_manager.cc:1428] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260810 18:50:24.485316 32179 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.487289 32179 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:24.490141 32071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:24.490793 32071 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 0.
I20260810 18:50:24.491847 31886 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters:
I20260810 18:50:24.492743 32179 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:24.493091 32179 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:24.493431 32179 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:24.500128 31805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:24.500093 32179 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.501210 31805 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 0.
I20260810 18:50:24.501029 31491 catalog_manager.cc:5495] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e reported cstate change: term changed from 0 to 1, leader changed from <none> to f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130). New cstate: current_term: 1 leader_uuid: "f58326f8fbd44e1984c2ce30da077d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:24.505941 32179 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 election: Requested vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:24.507498 31805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e"
I20260810 18:50:24.508134 31805 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:24.511986 32071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
I20260810 18:50:24.512560 32071 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:24.523383 32071 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 1.
I20260810 18:50:24.524680 31886 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters:
I20260810 18:50:24.525492 32179 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:24.526046 31805 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 1.
I20260810 18:50:24.526021 32179 raft_consensus.cc:686] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 1 LEADER]: Becoming Leader. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Running, Role: LEADER
I20260810 18:50:24.528909 32179 consensus_queue.cc:237] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:24.540284 31491 catalog_manager.cc:5495] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8314391cfbd645b4b780d02a43b4661d (127.28.93.131). New cstate: current_term: 1 leader_uuid: "8314391cfbd645b4b780d02a43b4661d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:24.541729 31492 catalog_manager.cc:5495] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e reported cstate change: term changed from 0 to 1, leader changed from <none> to f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130). New cstate: current_term: 1 leader_uuid: "f58326f8fbd44e1984c2ce30da077d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:24.564695 32179 consensus_queue.cc:1035] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:24.594357 32205 consensus_queue.cc:1035] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 18:50:24.603724 32179 consensus_queue.cc:1035] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260810 18:50:24.632344 32211 consensus_queue.cc:1035] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260810 18:50:24.645350 32179 consensus_queue.cc:1035] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260810 18:50:24.693199 32179 consensus_queue.cc:1035] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260810 18:50:25.713091 32241 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8314391cfbd645b4b780d02a43b4661d)
I20260810 18:50:25.731737 32241 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:25.753368 32205 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:25.755566 32241 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737), 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:25.787580 32205 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:25.789731 32205 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:25.790238 32205 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:25.790803 32205 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:25.793939 32205 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:25.832834 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:25.858441 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:25.859186 31804 raft_consensus.cc:2454] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 1.
W20260810 18:50:25.882014 32008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.93.130:45437, user_credentials={real_user=slave}} blocked reactor thread for 106623us
I20260810 18:50:25.885150 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:25.886734 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:25.888608 31806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:25.892127 32283 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 79547306558a49778d11f86a5276bb76)
I20260810 18:50:25.892997 32283 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:25.908708 32283 leader_election.cc:288] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7b3db47356949cc9049320effc90535 (127.28.93.190:38651), 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453)
I20260810 18:50:25.909826 31405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "f7b3db47356949cc9049320effc90535" is_pre_election: true
I20260810 18:50:25.916313 32282 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:25.917234 32282 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:25.924575 32281 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:25.926029 31886 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e; no voters: 8314391cfbd645b4b780d02a43b4661d
I20260810 18:50:25.925539 32281 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:25.927438 31804 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 1.
I20260810 18:50:25.928392 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:25.932569 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:25.940121 32241 raft_consensus.cc:2781] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:50:25.940663 32241 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8314391cfbd645b4b780d02a43b4661d)
I20260810 18:50:25.941030 32287 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 79547306558a49778d11f86a5276bb76)
I20260810 18:50:25.941429 32241 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:25.941885 32287 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:25.946969 32282 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:25.951750 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:25.952276 31510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7b3db47356949cc9049320effc90535" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "79547306558a49778d11f86a5276bb76" is_pre_election: true
I20260810 18:50:25.948836 32287 leader_election.cc:288] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453), bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699)
I20260810 18:50:25.954490 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7b3db47356949cc9049320effc90535" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" is_pre_election: true
I20260810 18:50:25.948684 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:25.961272 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:25.955440 32281 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:25.974931 32241 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:25.981009 31745 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e; no voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:25.983194 32281 raft_consensus.cc:2726] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:50:25.987896 32009 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d; no voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:25.988168 31510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "79547306558a49778d11f86a5276bb76" is_pre_election: true
I20260810 18:50:25.983157 31745 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e; no voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:25.991691 32009 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d; no voters: f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:25.994086 32281 raft_consensus.cc:2726] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:50:25.999869 32205 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:50:26.001169 32205 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Starting leader election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:26.001541 32205 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:26.001475 31568 leader_election.cc:302] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc51947e5d9c444e85b6c335c9cf79d8; no voters: 79547306558a49778d11f86a5276bb76, f7b3db47356949cc9049320effc90535
I20260810 18:50:26.002890 32283 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:50:26.005647 31367 leader_election.cc:302] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7b3db47356949cc9049320effc90535; no voters: 79547306558a49778d11f86a5276bb76, bc51947e5d9c444e85b6c335c9cf79d8
I20260810 18:50:26.006208 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e"
I20260810 18:50:26.010676 32287 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:50:26.002676 32286 raft_consensus.cc:2726] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:50:26.012637 32205 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.013650 32071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
I20260810 18:50:26.016987 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
I20260810 18:50:26.019953 32205 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 2 election: Requested vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:26.000451 32241 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 2 election: Requested vote from peers 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737), 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:26.031224 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e"
I20260810 18:50:26.037477 31886 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58326f8fbd44e1984c2ce30da077d2e; no voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d
I20260810 18:50:26.038548 32009 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d; no voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:26.039343 32205 raft_consensus.cc:2726] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:50:26.047094 32179 raft_consensus.cc:2726] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:50:26.062477 31938 raft_consensus.cc:1226] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Rejecting Update request from peer 8314391cfbd645b4b780d02a43b4661d for earlier term 1. Current term is 2. Ops: []
I20260810 18:50:26.069988 32205 consensus_queue.cc:1046] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:26.071426 32205 raft_consensus.cc:3032] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:26.082957 32205 raft_consensus.cc:729] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8314391cfbd645b4b780d02a43b4661d, State: Running, Role: LEADER
I20260810 18:50:26.088251 32205 consensus_queue.cc:260] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:26.090345 32205 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:26.101995 32069 raft_consensus.cc:1226] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Rejecting Update request from peer f58326f8fbd44e1984c2ce30da077d2e for earlier term 1. Current term is 2. Ops: []
I20260810 18:50:26.104350 32179 consensus_queue.cc:1046] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:26.105201 32179 raft_consensus.cc:3032] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:26.105450 32179 raft_consensus.cc:729] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Running, Role: LEADER
I20260810 18:50:26.106014 32179 consensus_queue.cc:260] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.107357 32179 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:26.120153 31938 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5"
dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
from {username='slave'} at 127.0.0.1:42670
I20260810 18:50:26.121122 31938 raft_consensus.cc:573] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Rejecting request to step down while not leader
I20260810 18:50:26.123441 29046 ts_tablet_manager-itest.cc:850] T:0f88e634539c49fd95f67aaedf60a5b5 P:f58326f8fbd44e1984c2ce30da077d2e: not a leader replica anymore
I20260810 18:50:26.125406 31938 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80"
dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
from {username='slave'} at 127.0.0.1:42670
I20260810 18:50:26.126044 31938 raft_consensus.cc:580] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 1 LEADER]: Received request to step down
I20260810 18:50:26.126435 31938 raft_consensus.cc:3032] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 1 LEADER]: Stepping down as leader of term 1
I20260810 18:50:26.126830 31938 raft_consensus.cc:729] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Running, Role: LEADER
I20260810 18:50:26.127826 31938 consensus_queue.cc:260] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.129194 31938 raft_consensus.cc:3037] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:26.253046 32281 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:26.253535 32281 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.255127 32281 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:26.256373 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:26.256778 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:26.257678 32069 raft_consensus.cc:2379] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 770b39bfac164d40a82408baae0ab88e in current term 2: Already voted for candidate 8314391cfbd645b4b780d02a43b4661d in this term.
I20260810 18:50:26.257328 31938 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770b39bfac164d40a82408baae0ab88e in term 2.
I20260810 18:50:26.258775 31744 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e; no voters: 8314391cfbd645b4b780d02a43b4661d
I20260810 18:50:26.258847 32281 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:26.259452 32282 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:50:26.259358 32281 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.259895 32282 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:26.260460 32282 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:26.260579 32179 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:26.261452 32281 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:26.261289 32179 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:26.262497 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:26.263249 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:26.264799 32071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:26.264983 32179 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737), 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:26.265658 32071 raft_consensus.cc:2454] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 2.
I20260810 18:50:26.267014 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:26.267421 31886 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters:
I20260810 18:50:26.269016 32179 raft_consensus.cc:2781] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:50:26.269536 32179 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:26.269860 32179 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:26.263394 31938 raft_consensus.cc:2454] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770b39bfac164d40a82408baae0ab88e in term 2.
I20260810 18:50:26.271581 31744 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e; no voters: 8314391cfbd645b4b780d02a43b4661d
I20260810 18:50:26.272979 32281 raft_consensus.cc:2781] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:50:26.273408 32281 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting leader election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:26.271759 32282 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.273823 32281 raft_consensus.cc:3037] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:26.278115 32282 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 election: Requested vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:26.280098 32071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
I20260810 18:50:26.280525 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
I20260810 18:50:26.280241 32179 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:26.281566 32071 raft_consensus.cc:2379] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 770b39bfac164d40a82408baae0ab88e in current term 2: Already voted for candidate 8314391cfbd645b4b780d02a43b4661d in this term.
I20260810 18:50:26.287043 32282 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8314391cfbd645b4b780d02a43b4661d)
I20260810 18:50:26.287210 32205 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:26.288002 32282 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:26.288017 32205 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.291633 32071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
I20260810 18:50:26.291499 32281 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.292550 32071 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:26.296703 32286 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:26.296960 31940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
I20260810 18:50:26.297358 32286 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.297964 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:26.298882 31804 raft_consensus.cc:2379] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8314391cfbd645b4b780d02a43b4661d in current term 2: Already voted for candidate 770b39bfac164d40a82408baae0ab88e in this term.
I20260810 18:50:26.298079 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
I20260810 18:50:26.299467 32069 raft_consensus.cc:3037] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:26.299855 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:26.300680 31937 raft_consensus.cc:2454] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 2.
I20260810 18:50:26.301214 32286 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:26.301075 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:26.303402 31804 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 2.
I20260810 18:50:26.304646 32009 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d; no voters:
I20260810 18:50:26.305910 32286 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:50:26.306066 32008 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters: 770b39bfac164d40a82408baae0ab88e
I20260810 18:50:26.303462 31940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:26.306519 31938 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770b39bfac164d40a82408baae0ab88e in term 2.
I20260810 18:50:26.297274 32281 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 election: Requested vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:26.307591 32286 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:26.308038 31744 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e; no voters: 8314391cfbd645b4b780d02a43b4661d
I20260810 18:50:26.309036 32281 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:26.308097 32286 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:26.311551 32071 raft_consensus.cc:2454] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 3.
I20260810 18:50:26.311807 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:26.312834 31938 raft_consensus.cc:2354] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 770b39bfac164d40a82408baae0ab88e for earlier term 2. Current term is 3.
I20260810 18:50:26.313154 31886 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters:
I20260810 18:50:26.310348 32205 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:26.313951 32238 raft_consensus.cc:2781] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:26.314579 32238 raft_consensus.cc:686] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 LEADER]: Becoming Leader. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Running, Role: LEADER
I20260810 18:50:26.310384 32281 raft_consensus.cc:686] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 2 LEADER]: Becoming Leader. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Running, Role: LEADER
I20260810 18:50:26.315186 31744 leader_election.cc:398] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Vote denied by peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437) with higher term. Message: Invalid argument: T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 770b39bfac164d40a82408baae0ab88e for earlier term 2. Current term is 3.
I20260810 18:50:26.315541 31744 leader_election.cc:401] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260810 18:50:26.313814 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e"
I20260810 18:50:26.316342 32069 raft_consensus.cc:2454] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770b39bfac164d40a82408baae0ab88e in term 2.
I20260810 18:50:26.316870 31804 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 1 FOLLOWER]: Advancing to term 3
I20260810 18:50:26.310956 32179 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 3 election: Requested vote from peers 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737), 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:26.315923 32281 consensus_queue.cc:237] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.318442 32238 consensus_queue.cc:237] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:26.316949 32285 raft_consensus.cc:2741] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260810 18:50:26.315542 32282 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:26.322352 31745 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d; no voters: f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:26.322916 32068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:26.323215 32282 raft_consensus.cc:2781] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:26.323758 32068 raft_consensus.cc:2354] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 770b39bfac164d40a82408baae0ab88e for earlier term 2. Current term is 3.
I20260810 18:50:26.328073 31804 raft_consensus.cc:2454] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 3.
I20260810 18:50:26.328671 32296 raft_consensus.cc:2726] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437) with higher term. Message: Invalid argument: T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 770b39bfac164d40a82408baae0ab88e for earlier term 2. Current term is 3.
I20260810 18:50:26.333367 31494 catalog_manager.cc:5495] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e reported cstate change: term changed from 1 to 3, leader changed from 8314391cfbd645b4b780d02a43b4661d (127.28.93.131) to f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130). New cstate: current_term: 3 leader_uuid: "f58326f8fbd44e1984c2ce30da077d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:26.337867 31745 leader_election.cc:398] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737) with higher term. Message: Invalid argument: T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 770b39bfac164d40a82408baae0ab88e for earlier term 2. Current term is 3.
I20260810 18:50:26.344322 31938 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f"
dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
from {username='slave'} at 127.0.0.1:42670
I20260810 18:50:26.345070 31938 raft_consensus.cc:580] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 LEADER]: Received request to step down
I20260810 18:50:26.343849 31481 catalog_manager.cc:5495] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e reported cstate change: term changed from 1 to 2, leader changed from f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130) to 770b39bfac164d40a82408baae0ab88e (127.28.93.129). New cstate: current_term: 2 leader_uuid: "770b39bfac164d40a82408baae0ab88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:26.345502 31938 raft_consensus.cc:3032] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:50:26.346117 32282 raft_consensus.cc:686] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 LEADER]: Becoming Leader. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Running, Role: LEADER
I20260810 18:50:26.352007 32282 consensus_queue.cc:237] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.348785 31938 raft_consensus.cc:729] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Running, Role: LEADER
I20260810 18:50:26.353861 31938 consensus_queue.cc:260] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:26.354938 31938 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.365554 32286 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.371832 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e"
I20260810 18:50:26.371909 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
I20260810 18:50:26.372860 31938 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:26.373617 32286 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 3 election: Requested vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:26.379818 31938 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 3.
I20260810 18:50:26.381556 32008 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters: 770b39bfac164d40a82408baae0ab88e
I20260810 18:50:26.385169 32286 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:26.385667 32286 raft_consensus.cc:686] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 3 LEADER]: Becoming Leader. State: Replica: 8314391cfbd645b4b780d02a43b4661d, State: Running, Role: LEADER
I20260810 18:50:26.386647 32286 consensus_queue.cc:237] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.421470 31938 raft_consensus.cc:1226] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Rejecting Update request from peer 770b39bfac164d40a82408baae0ab88e for earlier term 2. Current term is 3. Ops: []
I20260810 18:50:26.423842 32297 consensus_queue.cc:1046] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:26.425041 32297 raft_consensus.cc:3032] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 2 LEADER]: Stepping down as leader of term 2
I20260810 18:50:26.425501 32297 raft_consensus.cc:729] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Running, Role: LEADER
I20260810 18:50:26.425575 32071 raft_consensus.cc:1226] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 3 LEADER]: Rejecting Update request from peer 770b39bfac164d40a82408baae0ab88e for earlier term 2. Current term is 3. Ops: []
I20260810 18:50:26.426730 32297 consensus_queue.cc:260] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.428503 32297 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:26.429327 31492 catalog_manager.cc:5495] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d reported cstate change: term changed from 2 to 3, leader changed from 770b39bfac164d40a82408baae0ab88e (127.28.93.129) to 8314391cfbd645b4b780d02a43b4661d (127.28.93.131). New cstate: current_term: 3 leader_uuid: "8314391cfbd645b4b780d02a43b4661d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: HEALTHY } } }
W20260810 18:50:26.431380 32282 consensus_queue.cc:1159] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "8314391cfbd645b4b780d02a43b4661d" responder_term: 3 status { last_received { term: 3 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 770b39bfac164d40a82408baae0ab88e for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260810 18:50:26.460582 31492 catalog_manager.cc:5495] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e reported cstate change: term changed from 1 to 2, leader changed from f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130) to 770b39bfac164d40a82408baae0ab88e (127.28.93.129). New cstate: current_term: 2 leader_uuid: "770b39bfac164d40a82408baae0ab88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:26.468112 31804 raft_consensus.cc:1261] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Refusing update from remote peer 8314391cfbd645b4b780d02a43b4661d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20260810 18:50:26.468581 31804 pending_rounds.cc:76] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Aborting all ops after (but not including) 2
I20260810 18:50:26.468883 31804 pending_rounds.cc:97] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Aborting uncommitted NO_OP operation due to leader change: 2.3
I20260810 18:50:26.469300 31804 raft_consensus.cc:2866] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:50:26.471993 32286 consensus_queue.cc:1035] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:26.473039 31938 raft_consensus.cc:1261] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Refusing update from remote peer 8314391cfbd645b4b780d02a43b4661d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260810 18:50:26.478760 32286 consensus_queue.cc:1035] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:26.480767 32071 raft_consensus.cc:1261] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Refusing update from remote peer 770b39bfac164d40a82408baae0ab88e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 18:50:26.484079 31940 raft_consensus.cc:1261] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Refusing update from remote peer 770b39bfac164d40a82408baae0ab88e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 18:50:26.486199 32297 consensus_queue.cc:1035] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260810 18:50:26.492635 32297 consensus_queue.cc:1035] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:26.573974 32286 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:26.574849 32286 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:26.573109 32297 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:26.584929 32286 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:26.585626 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:26.586805 31938 raft_consensus.cc:2396] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8314391cfbd645b4b780d02a43b4661d for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260810 18:50:26.587970 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:26.579941 32297 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:26.590286 32297 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:26.590888 31804 raft_consensus.cc:2454] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 3.
I20260810 18:50:26.593521 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:26.595925 32009 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d; no voters: f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:26.596668 32286 raft_consensus.cc:2781] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:50:26.597045 32286 raft_consensus.cc:482] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:26.597442 32286 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.598079 32071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:26.597064 31938 raft_consensus.cc:2396] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 770b39bfac164d40a82408baae0ab88e for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260810 18:50:26.624071 32286 raft_consensus.cc:504] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:26.625139 32071 raft_consensus.cc:2379] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 770b39bfac164d40a82408baae0ab88e in current term 4: Already voted for candidate 8314391cfbd645b4b780d02a43b4661d in this term.
I20260810 18:50:26.625891 32286 leader_election.cc:288] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 4 election: Requested vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:26.626720 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
I20260810 18:50:26.627663 31938 raft_consensus.cc:2396] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 8314391cfbd645b4b780d02a43b4661d for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260810 18:50:26.627815 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e"
I20260810 18:50:26.628456 31804 raft_consensus.cc:3037] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.628883 32069 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5"
dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
from {username='slave'} at 127.0.0.1:39778
I20260810 18:50:26.629467 32069 raft_consensus.cc:580] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 3 LEADER]: Received request to step down
I20260810 18:50:26.629760 32069 raft_consensus.cc:3032] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 3 LEADER]: Stepping down as leader of term 3
I20260810 18:50:26.630044 32069 raft_consensus.cc:729] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8314391cfbd645b4b780d02a43b4661d, State: Running, Role: LEADER
I20260810 18:50:26.630700 32069 consensus_queue.cc:260] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.631845 32069 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.632539 31745 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e; no voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:26.634894 31804 raft_consensus.cc:2454] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 4.
I20260810 18:50:26.635275 32297 raft_consensus.cc:2726] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:50:26.635735 32009 leader_election.cc:302] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d; no voters: f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:26.636405 32286 raft_consensus.cc:2781] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:50:26.636788 32286 raft_consensus.cc:686] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 4 LEADER]: Becoming Leader. State: Replica: 8314391cfbd645b4b780d02a43b4661d, State: Running, Role: LEADER
I20260810 18:50:26.637531 32286 consensus_queue.cc:237] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:26.643196 31481 catalog_manager.cc:5495] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d reported cstate change: term changed from 3 to 4, leader changed from f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130) to 8314391cfbd645b4b780d02a43b4661d (127.28.93.131). New cstate: current_term: 4 leader_uuid: "8314391cfbd645b4b780d02a43b4661d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:26.743485 31938 raft_consensus.cc:1261] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Refusing update from remote peer 8314391cfbd645b4b780d02a43b4661d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260810 18:50:26.743808 31938 pending_rounds.cc:76] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Aborting all ops after (but not including) 2
I20260810 18:50:26.743978 31938 pending_rounds.cc:97] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260810 18:50:26.744139 31938 raft_consensus.cc:2866] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260810 18:50:26.745286 32286 consensus_queue.cc:1035] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:26.749266 32297 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8314391cfbd645b4b780d02a43b4661d)
I20260810 18:50:26.749780 32297 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.752802 32297 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:26.753476 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:26.755224 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:26.755769 31804 raft_consensus.cc:1261] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Refusing update from remote peer 8314391cfbd645b4b780d02a43b4661d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260810 18:50:26.755877 32069 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770b39bfac164d40a82408baae0ab88e in term 4.
I20260810 18:50:26.757496 31745 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d; no voters: f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:26.758965 32205 consensus_queue.cc:1035] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:26.759790 32297 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:50:26.760212 32297 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 8314391cfbd645b4b780d02a43b4661d)
I20260810 18:50:26.760593 32297 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.772511 32297 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.776371 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
I20260810 18:50:26.776573 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
I20260810 18:50:26.783558 32297 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 4 election: Requested vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:26.788139 32069 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770b39bfac164d40a82408baae0ab88e in term 4.
I20260810 18:50:26.793182 31745 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d; no voters: f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:26.795055 32297 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:50:26.796072 32297 raft_consensus.cc:686] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 4 LEADER]: Becoming Leader. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Running, Role: LEADER
I20260810 18:50:26.797361 32297 consensus_queue.cc:237] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.803335 32238 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 8314391cfbd645b4b780d02a43b4661d)
I20260810 18:50:26.803826 32238 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:26.806360 32238 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:26.807870 31804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
I20260810 18:50:26.807927 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:26.809543 32069 raft_consensus.cc:2379] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f58326f8fbd44e1984c2ce30da077d2e in current term 4: Already voted for candidate 770b39bfac164d40a82408baae0ab88e in this term.
I20260810 18:50:26.810612 31886 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58326f8fbd44e1984c2ce30da077d2e; no voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d
I20260810 18:50:26.811967 32238 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:26.810474 31492 catalog_manager.cc:5495] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e reported cstate change: term changed from 3 to 4, leader changed from 8314391cfbd645b4b780d02a43b4661d (127.28.93.131) to 770b39bfac164d40a82408baae0ab88e (127.28.93.129). New cstate: current_term: 4 leader_uuid: "770b39bfac164d40a82408baae0ab88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:26.820842 32238 raft_consensus.cc:2726] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:50:26.837597 31481 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51686:
table { table_id: "c384a89a9c1b42f8ae3b3f1d6815aba9" table_name: "test-table" } new_table_name: "NewTableName"
I20260810 18:50:26.877988 32069 raft_consensus.cc:1261] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Refusing update from remote peer 770b39bfac164d40a82408baae0ab88e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260810 18:50:26.879768 31938 raft_consensus.cc:1261] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Refusing update from remote peer 770b39bfac164d40a82408baae0ab88e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260810 18:50:26.885298 32296 consensus_queue.cc:1035] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260810 18:50:26.901974 32296 consensus_queue.cc:1035] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260810 18:50:26.911727 32331 tablet.cc:1722] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:26.917510 29046 master.cc:537] Master@127.28.93.189:43453 shutting down...
I20260810 18:50:26.924633 32334 tablet.cc:1722] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:26.998067 31492 catalog_manager.cc:5587] TS 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737) does not have the latest schema for tablet 618c53eed4a94a9eab8d7c30da86b31f (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9]). Expected version 1 got 0
I20260810 18:50:27.003449 32339 tablet.cc:1722] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:27.024226 32337 tablet.cc:1722] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:27.016608 32335 tablet.cc:1722] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:27.026422 32338 tablet.cc:1722] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:27.041054 32340 tablet.cc:1722] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:27.058207 32339 tablet.cc:1722] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:27.061559 32341 tablet.cc:1722] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:27.143062 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:27.143872 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:27.144254 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: stopping tablet replica
I20260810 18:50:27.166030 29046 master.cc:552] Master@127.28.93.189:43453 shutdown complete.
I20260810 18:50:27.270709 32347 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 79547306558a49778d11f86a5276bb76)
I20260810 18:50:27.271109 32347 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:27.272575 32347 leader_election.cc:288] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7b3db47356949cc9049320effc90535 (127.28.93.190:38651), 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453)
I20260810 18:50:27.273386 31405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "f7b3db47356949cc9049320effc90535" is_pre_election: true
W20260810 18:50:27.274783 31568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.93.189:43453: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20260810 18:50:27.277469 31568 leader_election.cc:334] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453): Network error: Client connection negotiation failed: client connection to 127.28.93.189:43453: connect: Connection refused (error 111)
I20260810 18:50:27.277771 31568 leader_election.cc:302] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc51947e5d9c444e85b6c335c9cf79d8; no voters: 79547306558a49778d11f86a5276bb76, f7b3db47356949cc9049320effc90535
I20260810 18:50:27.278293 32347 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:50:27.345636 32352 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 79547306558a49778d11f86a5276bb76)
I20260810 18:50:27.346166 32352 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:27.348243 32352 leader_election.cc:288] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453), bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699)
I20260810 18:50:27.349813 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7b3db47356949cc9049320effc90535" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" is_pre_election: true
I20260810 18:50:27.350447 31604 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7b3db47356949cc9049320effc90535 in term 1.
I20260810 18:50:27.351920 31367 leader_election.cc:302] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc51947e5d9c444e85b6c335c9cf79d8, f7b3db47356949cc9049320effc90535; no voters:
I20260810 18:50:27.352705 32352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:50:27.353078 32352 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 79547306558a49778d11f86a5276bb76)
I20260810 18:50:27.353444 32352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:50:27.355422 31372 leader_election.cc:334] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453): Network error: Client connection negotiation failed: client connection to 127.28.93.189:43453: connect: Connection refused (error 111)
I20260810 18:50:27.358881 32352 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:27.360354 32352 leader_election.cc:288] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 2 election: Requested vote from peers 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453), bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699)
I20260810 18:50:27.361181 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7b3db47356949cc9049320effc90535" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "bc51947e5d9c444e85b6c335c9cf79d8"
I20260810 18:50:27.361632 31604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:50:27.362957 31372 leader_election.cc:334] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453): Network error: Client connection negotiation failed: client connection to 127.28.93.189:43453: connect: Connection refused (error 111)
I20260810 18:50:27.366506 31604 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7b3db47356949cc9049320effc90535 in term 2.
I20260810 18:50:27.367354 31367 leader_election.cc:302] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bc51947e5d9c444e85b6c335c9cf79d8, f7b3db47356949cc9049320effc90535; no voters: 79547306558a49778d11f86a5276bb76
I20260810 18:50:27.367961 32352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:27.368786 32352 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 2 LEADER]: Becoming Leader. State: Replica: f7b3db47356949cc9049320effc90535, State: Running, Role: LEADER
I20260810 18:50:27.369391 32352 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:27.372982 32357 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b3db47356949cc9049320effc90535. Latest consensus state: current_term: 2 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:27.373522 32357 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:27.374629 32358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:27.379884 32358 catalog_manager.cc:669] Loaded metadata for table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9]
I20260810 18:50:27.381038 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:50:27.387419 32358 tablet_loader.cc:96] loaded metadata for tablet 0f88e634539c49fd95f67aaedf60a5b5 (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9])
W20260810 18:50:27.387755 32359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:27.388219 32360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:27.389138 32358 tablet_loader.cc:96] loaded metadata for tablet 5f0d266df91e4d60b5289e0683068f80 (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9])
I20260810 18:50:27.390465 32358 tablet_loader.cc:96] loaded metadata for tablet 618c53eed4a94a9eab8d7c30da86b31f (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9])
W20260810 18:50:27.390828 32364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:27.391431 32358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:27.391427 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:27.392763 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:27.392992 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:27.393154 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387827393142 us; error 0 us; skew 500 ppm
I20260810 18:50:27.393744 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:27.395582 32358 catalog_manager.cc:1259] Loaded cluster ID: c3bf6b332118469eb0628784c9263293
I20260810 18:50:27.395925 32358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:27.398790 29046 webserver.cc:460] Webserver started at http://127.28.93.189:43141/ using document root <none> and password file <none>
I20260810 18:50:27.399493 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:27.399761 29046 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260810 18:50:27.402259 32358 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:27.405987 29046 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 18:50:27.406318 32358 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: Loaded TSK: 0
I20260810 18:50:27.407702 32358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:27.409230 32371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:27.410099 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:27.410374 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-1-root
uuid: "79547306558a49778d11f86a5276bb76"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:27.410722 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:27.463495 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:27.464942 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:27.472127 31604 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 2 FOLLOWER]: Refusing update from remote peer f7b3db47356949cc9049320effc90535: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20260810 18:50:27.473219 32357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260810 18:50:27.477082 32347 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b3db47356949cc9049320effc90535. Latest consensus state: current_term: 2 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:27.478047 32347 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 18:50:27.484791 31372 consensus_peers.cc:487] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 -> Peer 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453): Couldn't send request to peer 79547306558a49778d11f86a5276bb76. Status: Network error: Client connection negotiation failed: client connection to 127.28.93.189:43453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:27.491001 32347 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:27.491861 32347 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:27.496562 32352 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:27.497318 32352 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:27.519007 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.189:43453
I20260810 18:50:27.519156 32469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.189:43453 every 8 connection(s)
I20260810 18:50:27.526734 31982 heartbeater.cc:503] Master 127.28.93.190:38651 was elected leader, sending a full tablet report...
I20260810 18:50:27.534125 32471 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:50:27.536255 31978 heartbeater.cc:347] Connected to a master server at 127.28.93.189:43453
I20260810 18:50:27.541360 32471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: Bootstrap starting.
I20260810 18:50:27.550475 31861 heartbeater.cc:503] Master 127.28.93.190:38651 was elected leader, sending a full tablet report...
I20260810 18:50:27.555876 32120 heartbeater.cc:503] Master 127.28.93.190:38651 was elected leader, sending a full tablet report...
I20260810 18:50:27.573830 31388 catalog_manager.cc:6040] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9] alter complete (version 1)
I20260810 18:50:27.597084 31859 heartbeater.cc:347] Connected to a master server at 127.28.93.189:43453
I20260810 18:50:27.610914 32116 heartbeater.cc:347] Connected to a master server at 127.28.93.189:43453
I20260810 18:50:27.614724 32471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=7 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:27.615873 32471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: Bootstrap complete.
I20260810 18:50:27.618364 32471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:27.618899 32471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79547306558a49778d11f86a5276bb76, State: Initialized, Role: FOLLOWER
I20260810 18:50:27.619594 32471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:27.621402 32479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:27.621997 32479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:27.622846 32471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:27.623238 32428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:27.630045 29046 master.cc:537] Master@127.28.93.190:38651 shutting down...
I20260810 18:50:27.689713 32490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:27.690579 32490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:27.693689 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 2 LEADER]: Raft consensus shutting down.
I20260810 18:50:27.691718 32479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b3db47356949cc9049320effc90535. Latest consensus state: current_term: 2 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:27.695621 32479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:27.696063 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:27.696609 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: stopping tablet replica
I20260810 18:50:27.704509 32504 catalog_manager.cc:1259] Loaded cluster ID: c3bf6b332118469eb0628784c9263293
I20260810 18:50:27.704733 32504 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: loading cluster ID for follower catalog manager: success
I20260810 18:50:27.710078 32504 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: acquiring CA information for follower catalog manager: success
I20260810 18:50:27.713948 32504 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:50:27.738485 29046 master.cc:552] Master@127.28.93.190:38651 shutdown complete.
I20260810 18:50:27.754894 32404 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" instance_seqno: 1786387823382438) as {username='slave'} at 127.0.0.1:51854; Asking this server to re-register.
I20260810 18:50:27.756098 32116 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:27.758610 32404 ts_manager.cc:194] Registered new tserver with Master: 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:27.773723 32404 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" instance_seqno: 1786387823020248) as {username='slave'} at 127.0.0.1:51828; Asking this server to re-register.
I20260810 18:50:27.775876 31978 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:27.775897 32406 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "770b39bfac164d40a82408baae0ab88e" instance_seqno: 1786387822690778) as {username='slave'} at 127.0.0.1:51834; Asking this server to re-register.
I20260810 18:50:27.777374 31859 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:27.778973 32406 ts_manager.cc:194] Registered new tserver with Master: f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:27.780218 32404 ts_manager.cc:194] Registered new tserver with Master: 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:27.838778 32479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7b3db47356949cc9049320effc90535)
I20260810 18:50:27.839280 32479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:27.841593 32479 leader_election.cc:288] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7b3db47356949cc9049320effc90535 (127.28.93.190:38651), bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699)
W20260810 18:50:27.846988 32376 leader_election.cc:334] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f7b3db47356949cc9049320effc90535 (127.28.93.190:38651): Network error: Client connection negotiation failed: client connection to 127.28.93.190:38651: connect: Connection refused (error 111)
I20260810 18:50:27.854584 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79547306558a49778d11f86a5276bb76" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" is_pre_election: true
I20260810 18:50:27.855589 32376 leader_election.cc:302] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79547306558a49778d11f86a5276bb76; no voters: bc51947e5d9c444e85b6c335c9cf79d8, f7b3db47356949cc9049320effc90535
I20260810 18:50:27.856225 32479 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:50:27.858253 32347 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f7b3db47356949cc9049320effc90535)
I20260810 18:50:27.858626 32347 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:27.860172 32347 leader_election.cc:288] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f7b3db47356949cc9049320effc90535 (127.28.93.190:38651), 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453)
W20260810 18:50:27.865082 31566 leader_election.cc:334] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f7b3db47356949cc9049320effc90535 (127.28.93.190:38651): Network error: Client connection negotiation failed: client connection to 127.28.93.190:38651: connect: Connection refused (error 111)
I20260810 18:50:27.871820 32428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "79547306558a49778d11f86a5276bb76" is_pre_election: true
I20260810 18:50:27.872330 32428 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc51947e5d9c444e85b6c335c9cf79d8 in term 2.
I20260810 18:50:27.873092 31568 leader_election.cc:302] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79547306558a49778d11f86a5276bb76, bc51947e5d9c444e85b6c335c9cf79d8; no voters: f7b3db47356949cc9049320effc90535
I20260810 18:50:27.873799 32347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:50:27.874172 32347 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f7b3db47356949cc9049320effc90535)
I20260810 18:50:27.874526 32347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:27.879560 32347 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:27.881232 32347 leader_election.cc:288] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 3 election: Requested vote from peers f7b3db47356949cc9049320effc90535 (127.28.93.190:38651), 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453)
I20260810 18:50:27.882799 32428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "79547306558a49778d11f86a5276bb76"
I20260810 18:50:27.884135 32428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 2 FOLLOWER]: Advancing to term 3
W20260810 18:50:27.886826 31566 leader_election.cc:334] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f7b3db47356949cc9049320effc90535 (127.28.93.190:38651): Network error: Client connection negotiation failed: client connection to 127.28.93.190:38651: connect: Connection refused (error 111)
I20260810 18:50:27.889123 32428 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc51947e5d9c444e85b6c335c9cf79d8 in term 3.
I20260810 18:50:27.890019 31568 leader_election.cc:302] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79547306558a49778d11f86a5276bb76, bc51947e5d9c444e85b6c335c9cf79d8; no voters: f7b3db47356949cc9049320effc90535
I20260810 18:50:27.890820 32347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:27.891937 32347 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 3 LEADER]: Becoming Leader. State: Replica: bc51947e5d9c444e85b6c335c9cf79d8, State: Running, Role: LEADER
I20260810 18:50:27.892570 32347 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:27.895105 32516 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc51947e5d9c444e85b6c335c9cf79d8. Latest consensus state: current_term: 3 leader_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:27.895738 32516 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:27.897439 32518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:27.902220 32518 catalog_manager.cc:669] Loaded metadata for table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9]
I20260810 18:50:27.905931 32518 tablet_loader.cc:96] loaded metadata for tablet 0f88e634539c49fd95f67aaedf60a5b5 (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9])
I20260810 18:50:27.907116 32518 tablet_loader.cc:96] loaded metadata for tablet 5f0d266df91e4d60b5289e0683068f80 (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9])
I20260810 18:50:27.908286 32518 tablet_loader.cc:96] loaded metadata for tablet 618c53eed4a94a9eab8d7c30da86b31f (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9])
I20260810 18:50:27.909639 32518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:27.912897 32518 catalog_manager.cc:1259] Loaded cluster ID: c3bf6b332118469eb0628784c9263293
I20260810 18:50:27.913085 32518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:27.916818 32518 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:27.919754 32518 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: Loaded TSK: 0
I20260810 18:50:27.920611 32518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:27.953245 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:27.958289 32519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:27.959213 32521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:27.961112 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:27.961591 32523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:27.962476 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:27.962684 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:27.962844 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387827962834 us; error 0 us; skew 500 ppm
I20260810 18:50:27.963339 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:27.966038 29046 webserver.cc:460] Webserver started at http://127.28.93.190:33385/ using document root <none> and password file <none>
I20260810 18:50:27.966485 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:27.966650 29046 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260810 18:50:27.970587 29046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:50:27.973434 32529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:27.974159 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:50:27.974438 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "f7b3db47356949cc9049320effc90535"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:27.974702 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:27.996444 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:27.997720 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260810 18:50:28.006182 31566 consensus_peers.cc:487] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 -> Peer f7b3db47356949cc9049320effc90535 (127.28.93.190:38651): Couldn't send request to peer f7b3db47356949cc9049320effc90535. Status: Network error: Client connection negotiation failed: client connection to 127.28.93.190:38651: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:28.017267 32428 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 3 FOLLOWER]: Refusing update from remote peer bc51947e5d9c444e85b6c335c9cf79d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20260810 18:50:28.018944 32516 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260810 18:50:28.024260 32479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc51947e5d9c444e85b6c335c9cf79d8. Latest consensus state: current_term: 3 leader_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.025086 32479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:28.028594 32479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.029405 32479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:28.030687 32516 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.031474 32516 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:28.043730 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:38651
I20260810 18:50:28.043830 32611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:38651 every 8 connection(s)
I20260810 18:50:28.052838 32612 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:50:28.055907 32115 heartbeater.cc:503] Master 127.28.93.188:36699 was elected leader, sending a full tablet report...
I20260810 18:50:28.057427 32612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: Bootstrap starting.
I20260810 18:50:28.089572 31855 heartbeater.cc:503] Master 127.28.93.188:36699 was elected leader, sending a full tablet report...
I20260810 18:50:28.092730 31977 heartbeater.cc:503] Master 127.28.93.188:36699 was elected leader, sending a full tablet report...
I20260810 18:50:28.098448 32120 heartbeater.cc:347] Connected to a master server at 127.28.93.190:38651
I20260810 18:50:28.109727 31982 heartbeater.cc:347] Connected to a master server at 127.28.93.190:38651
I20260810 18:50:28.116760 31861 heartbeater.cc:347] Connected to a master server at 127.28.93.190:38651
I20260810 18:50:28.140471 32612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=7 ignored=0} mutations{seen=14 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:28.141249 32612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: Bootstrap complete.
I20260810 18:50:28.143389 32612 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:28.143973 32612 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b3db47356949cc9049320effc90535, State: Initialized, Role: FOLLOWER
I20260810 18:50:28.144542 32612 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:28.146059 32623 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.146837 32623 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:28.146979 32612 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:28.150458 29046 master.cc:537] Master@127.28.93.188:36699 shutting down...
I20260810 18:50:28.156167 32575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:28.189553 32645 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.190428 32645 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:28.190976 32623 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc51947e5d9c444e85b6c335c9cf79d8. Latest consensus state: current_term: 3 leader_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.191551 32623 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:28.197151 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:50:28.198040 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:28.198567 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: stopping tablet replica
I20260810 18:50:28.212706 32648 catalog_manager.cc:1259] Loaded cluster ID: c3bf6b332118469eb0628784c9263293
I20260810 18:50:28.212983 32648 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: loading cluster ID for follower catalog manager: success
I20260810 18:50:28.219906 32648 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: acquiring CA information for follower catalog manager: success
I20260810 18:50:28.223922 32648 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:50:28.233251 29046 master.cc:552] Master@127.28.93.188:36699 shutdown complete.
I20260810 18:50:28.258596 32560 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" instance_seqno: 1786387823020248) as {username='slave'} at 127.0.0.1:59504; Asking this server to re-register.
I20260810 18:50:28.259714 31982 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:28.262945 32560 ts_manager.cc:194] Registered new tserver with Master: f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:28.271063 32560 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" instance_seqno: 1786387823382438) as {username='slave'} at 127.0.0.1:59496; Asking this server to re-register.
I20260810 18:50:28.274135 32120 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:28.276801 32560 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "770b39bfac164d40a82408baae0ab88e" instance_seqno: 1786387822690778) as {username='slave'} at 127.0.0.1:59518; Asking this server to re-register.
I20260810 18:50:28.278851 32562 ts_manager.cc:194] Registered new tserver with Master: 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:28.280974 31861 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:28.284106 32562 ts_manager.cc:194] Registered new tserver with Master: 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:28.313773 32479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bc51947e5d9c444e85b6c335c9cf79d8)
I20260810 18:50:28.314213 32479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:28.315888 32479 leader_election.cc:288] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f7b3db47356949cc9049320effc90535 (127.28.93.190:38651), bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699)
W20260810 18:50:28.319330 32376 leader_election.cc:334] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699): Network error: Client connection negotiation failed: client connection to 127.28.93.188:36699: connect: Connection refused (error 111)
I20260810 18:50:28.325014 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79547306558a49778d11f86a5276bb76" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "f7b3db47356949cc9049320effc90535" is_pre_election: true
I20260810 18:50:28.326141 32376 leader_election.cc:302] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79547306558a49778d11f86a5276bb76; no voters: bc51947e5d9c444e85b6c335c9cf79d8, f7b3db47356949cc9049320effc90535
I20260810 18:50:28.326787 32479 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260810 18:50:28.374286 32623 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bc51947e5d9c444e85b6c335c9cf79d8)
I20260810 18:50:28.374811 32623 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:28.377135 32623 leader_election.cc:288] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453), bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699)
W20260810 18:50:28.385071 32531 leader_election.cc:334] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699): Network error: Client connection negotiation failed: client connection to 127.28.93.188:36699: connect: Connection refused (error 111)
I20260810 18:50:28.391633 32428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7b3db47356949cc9049320effc90535" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "79547306558a49778d11f86a5276bb76" is_pre_election: true
I20260810 18:50:28.392149 32428 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7b3db47356949cc9049320effc90535 in term 3.
I20260810 18:50:28.392877 32535 leader_election.cc:302] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79547306558a49778d11f86a5276bb76, f7b3db47356949cc9049320effc90535; no voters: bc51947e5d9c444e85b6c335c9cf79d8
I20260810 18:50:28.393481 32623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260810 18:50:28.393759 32623 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bc51947e5d9c444e85b6c335c9cf79d8)
I20260810 18:50:28.394021 32623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:28.398411 32623 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:28.399811 32623 leader_election.cc:288] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 4 election: Requested vote from peers 79547306558a49778d11f86a5276bb76 (127.28.93.189:43453), bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699)
I20260810 18:50:28.400404 32428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7b3db47356949cc9049320effc90535" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "79547306558a49778d11f86a5276bb76"
I20260810 18:50:28.401027 32428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 3 FOLLOWER]: Advancing to term 4
W20260810 18:50:28.402980 32531 leader_election.cc:334] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699): Network error: Client connection negotiation failed: client connection to 127.28.93.188:36699: connect: Connection refused (error 111)
I20260810 18:50:28.405134 32428 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7b3db47356949cc9049320effc90535 in term 4.
I20260810 18:50:28.405941 32535 leader_election.cc:302] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79547306558a49778d11f86a5276bb76, f7b3db47356949cc9049320effc90535; no voters: bc51947e5d9c444e85b6c335c9cf79d8
I20260810 18:50:28.406513 32623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 4 FOLLOWER]: Leader election won for term 4
I20260810 18:50:28.406877 32623 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 4 LEADER]: Becoming Leader. State: Replica: f7b3db47356949cc9049320effc90535, State: Running, Role: LEADER
I20260810 18:50:28.407543 32623 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 3.19, Last appended by leader: 19, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:28.410008 32645 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b3db47356949cc9049320effc90535. Latest consensus state: current_term: 4 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.410537 32645 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:28.412075 32655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:28.416332 32655 catalog_manager.cc:669] Loaded metadata for table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9]
I20260810 18:50:28.420837 32655 tablet_loader.cc:96] loaded metadata for tablet 0f88e634539c49fd95f67aaedf60a5b5 (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9])
I20260810 18:50:28.421828 32655 tablet_loader.cc:96] loaded metadata for tablet 5f0d266df91e4d60b5289e0683068f80 (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9])
I20260810 18:50:28.422745 32655 tablet_loader.cc:96] loaded metadata for tablet 618c53eed4a94a9eab8d7c30da86b31f (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9])
I20260810 18:50:28.423620 32655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:28.426766 32655 catalog_manager.cc:1259] Loaded cluster ID: c3bf6b332118469eb0628784c9263293
I20260810 18:50:28.426962 32655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:28.430689 32655 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:28.433653 32655 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535: Loaded TSK: 0
I20260810 18:50:28.434526 32655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:28.446535 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:28.451737 32657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:28.452967 32658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:28.454247 32663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:28.454324 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:28.455329 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:28.455585 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:28.455816 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387828455798 us; error 0 us; skew 500 ppm
I20260810 18:50:28.456434 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:28.459007 29046 webserver.cc:460] Webserver started at http://127.28.93.188:45095/ using document root <none> and password file <none>
I20260810 18:50:28.459447 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:28.459612 29046 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260810 18:50:28.462809 29046 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:28.465910 32671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:28.466751 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:50:28.467089 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-2-root
uuid: "bc51947e5d9c444e85b6c335c9cf79d8"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:28.467406 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:28.480055 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:28.481482 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:28.500067 32428 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 4 FOLLOWER]: Refusing update from remote peer f7b3db47356949cc9049320effc90535: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 4 index: 20. (index mismatch)
I20260810 18:50:28.501583 32645 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260810 18:50:28.504784 32479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b3db47356949cc9049320effc90535. Latest consensus state: current_term: 4 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.505587 32479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:28.508405 32719 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.509156 32719 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:28.511554 32479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
W20260810 18:50:28.511950 32531 consensus_peers.cc:487] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 -> Peer bc51947e5d9c444e85b6c335c9cf79d8 (127.28.93.188:36699): Couldn't send request to peer bc51947e5d9c444e85b6c335c9cf79d8. Status: Network error: Client connection negotiation failed: client connection to 127.28.93.188:36699: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:28.512331 32479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:28.529682 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.188:36699
I20260810 18:50:28.529830 32758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.188:36699 every 8 connection(s)
I20260810 18:50:28.538578 32759 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:50:28.542769 32759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: Bootstrap starting.
I20260810 18:50:28.573040 31982 heartbeater.cc:503] Master 127.28.93.190:38651 was elected leader, sending a full tablet report...
I20260810 18:50:28.590085 32120 heartbeater.cc:503] Master 127.28.93.190:38651 was elected leader, sending a full tablet report...
I20260810 18:50:28.590988 32115 heartbeater.cc:347] Connected to a master server at 127.28.93.188:36699
I20260810 18:50:28.592409 31855 heartbeater.cc:347] Connected to a master server at 127.28.93.188:36699
I20260810 18:50:28.592603 31861 heartbeater.cc:503] Master 127.28.93.190:38651 was elected leader, sending a full tablet report...
I20260810 18:50:28.603549 31977 heartbeater.cc:347] Connected to a master server at 127.28.93.188:36699
I20260810 18:50:28.613768 32759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=7 ignored=0} mutations{seen=14 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:28.614580 32759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: Bootstrap complete.
I20260810 18:50:28.617092 32759 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:28.617849 32759 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc51947e5d9c444e85b6c335c9cf79d8, State: Initialized, Role: FOLLOWER
I20260810 18:50:28.618408 32759 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 3.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } }
I20260810 18:50:28.620234 32766 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.620786 32766 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:28.621810 32759 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:28.622190 32720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 3 FOLLOWER]: Advancing to term 4
I20260810 18:50:28.634186 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
W20260810 18:50:28.674271 32009 consensus_peers.cc:487] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d -> Peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Couldn't send request to peer 770b39bfac164d40a82408baae0ab88e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:28.675814 32766 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b3db47356949cc9049320effc90535. Latest consensus state: current_term: 4 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.675420 323 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.676648 32766 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:28.676733 323 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:50:28.688802 325 catalog_manager.cc:1259] Loaded cluster ID: c3bf6b332118469eb0628784c9263293
I20260810 18:50:28.689139 325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: loading cluster ID for follower catalog manager: success
I20260810 18:50:28.697621 325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: acquiring CA information for follower catalog manager: success
I20260810 18:50:28.701588 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:28.702353 29046 tablet_replica.cc:330] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: stopping tablet replica
I20260810 18:50:28.703078 29046 raft_consensus.cc:2229] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 4 LEADER]: Raft consensus shutting down.
I20260810 18:50:28.703234 325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:50:28.704042 29046 raft_consensus.cc:2258] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:28.705967 29046 tablet_replica.cc:330] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: stopping tablet replica
I20260810 18:50:28.706400 29046 raft_consensus.cc:2229] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:28.706736 29046 raft_consensus.cc:2258] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:28.708592 29046 tablet_replica.cc:330] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: stopping tablet replica
I20260810 18:50:28.709093 29046 raft_consensus.cc:2229] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 LEADER]: Raft consensus shutting down.
I20260810 18:50:28.709940 29046 raft_consensus.cc:2258] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:28.710073 32645 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "f7b3db47356949cc9049320effc90535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7b3db47356949cc9049320effc90535" member_type: VOTER last_known_addr { host: "127.28.93.190" port: 38651 } } peers { permanent_uuid: "79547306558a49778d11f86a5276bb76" member_type: VOTER last_known_addr { host: "127.28.93.189" port: 43453 } } peers { permanent_uuid: "bc51947e5d9c444e85b6c335c9cf79d8" member_type: VOTER last_known_addr { host: "127.28.93.188" port: 36699 } } }
I20260810 18:50:28.710793 32645 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:28.731659 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:28.744719 32698 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" instance_seqno: 1786387823382438) as {username='slave'} at 127.0.0.1:44300; Asking this server to re-register.
I20260810 18:50:28.746362 32115 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:28.748524 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:50:28.749203 32698 ts_manager.cc:194] Registered new tserver with Master: 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
W20260810 18:50:28.754345 333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:28.755111 334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:28.756614 338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:28.759377 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:28.759775 32698 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" instance_seqno: 1786387823020248) as {username='slave'} at 127.0.0.1:44306; Asking this server to re-register.
I20260810 18:50:28.760949 31977 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:28.761518 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:28.761723 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:28.761893 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387828761879 us; error 0 us; skew 500 ppm
I20260810 18:50:28.762409 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:28.763846 32698 ts_manager.cc:194] Registered new tserver with Master: f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:28.765208 29046 webserver.cc:460] Webserver started at http://127.28.93.129:43079/ using document root <none> and password file <none>
I20260810 18:50:28.765653 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:28.765790 29046 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260810 18:50:28.769366 29046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:50:28.772667 346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:28.773396 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:28.773671 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "770b39bfac164d40a82408baae0ab88e"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:28.773929 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:28.785377 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:28.786429 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:28.787745 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:28.790961 362 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20260810 18:50:28.805052 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20260810 18:50:28.805377 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260810 18:50:28.805728 29046 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20260810 18:50:28.810953 362 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Bootstrap starting.
I20260810 18:50:28.820755 29046 ts_tablet_manager.cc:610] Registered 3 tablets
I20260810 18:50:28.821066 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.015s user 0.012s sys 0.002s
I20260810 18:50:28.826134 328 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 770b39bfac164d40a82408baae0ab88e)
I20260810 18:50:28.826658 328 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:28.828641 328 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:28.830207 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
W20260810 18:50:28.834239 32009 leader_election.cc:334] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Network error: Client connection negotiation failed: client connection to 127.28.93.129:43783: connect: Connection refused (error 111)
I20260810 18:50:28.834662 32009 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d; no voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:28.835391 328 raft_consensus.cc:2726] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260810 18:50:28.856408 396 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 770b39bfac164d40a82408baae0ab88e)
I20260810 18:50:28.856963 396 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:28.861516 406 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 770b39bfac164d40a82408baae0ab88e)
I20260810 18:50:28.862684 406 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:28.864682 406 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:28.866623 396 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:28.867198 32069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:28.869187 32069 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 4.
I20260810 18:50:28.870407 32068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:28.876056 31886 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters:
I20260810 18:50:28.877466 396 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260810 18:50:28.877847 396 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 770b39bfac164d40a82408baae0ab88e)
W20260810 18:50:28.878087 31886 leader_election.cc:334] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Network error: Client connection negotiation failed: client connection to 127.28.93.129:43783: connect: Connection refused (error 111)
I20260810 18:50:28.878211 396 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:50:28.878813 31886 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f58326f8fbd44e1984c2ce30da077d2e; no voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d
I20260810 18:50:28.879647 406 raft_consensus.cc:2726] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260810 18:50:28.880592 31886 leader_election.cc:334] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Network error: Client connection negotiation failed: client connection to 127.28.93.129:43783: connect: Connection refused (error 111)
I20260810 18:50:28.884922 396 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:28.886557 396 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 5 election: Requested vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:28.888391 328 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 770b39bfac164d40a82408baae0ab88e)
I20260810 18:50:28.888860 328 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:28.890751 328 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
W20260810 18:50:28.890712 31886 leader_election.cc:334] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Network error: Client connection negotiation failed: client connection to 127.28.93.129:43783: connect: Connection refused (error 111)
I20260810 18:50:28.891640 32068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "f58326f8fbd44e1984c2ce30da077d2e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
I20260810 18:50:28.892259 32068 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:50:28.893196 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:28.893810 31937 raft_consensus.cc:2454] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 2.
I20260810 18:50:28.895088 32008 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters:
I20260810 18:50:28.896651 328 raft_consensus.cc:2781] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:50:28.897013 328 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 770b39bfac164d40a82408baae0ab88e)
I20260810 18:50:28.897334 328 raft_consensus.cc:3037] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:28.899564 32068 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f58326f8fbd44e1984c2ce30da077d2e in term 5.
I20260810 18:50:28.900653 31886 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters: 770b39bfac164d40a82408baae0ab88e
W20260810 18:50:28.902143 32009 leader_election.cc:334] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Network error: Client connection negotiation failed: client connection to 127.28.93.129:43783: connect: Connection refused (error 111)
I20260810 18:50:28.903142 396 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 5 FOLLOWER]: Leader election won for term 5
I20260810 18:50:28.903640 396 raft_consensus.cc:686] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 5 LEADER]: Becoming Leader. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Running, Role: LEADER
I20260810 18:50:28.905385 328 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:28.906343 396 consensus_queue.cc:237] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:28.907998 362 tablet.cc:1722] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:28.908820 362 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20260810 18:50:28.910223 362 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=266 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:28.911212 362 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Bootstrap complete.
W20260810 18:50:28.911592 32009 leader_election.cc:334] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Network error: Client connection negotiation failed: client connection to 127.28.93.129:43783: connect: Connection refused (error 111)
I20260810 18:50:28.913724 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e"
I20260810 18:50:28.914319 31937 raft_consensus.cc:3037] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:28.925460 31937 raft_consensus.cc:2454] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8314391cfbd645b4b780d02a43b4661d in term 3.
I20260810 18:50:28.927237 32008 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e; no voters: 770b39bfac164d40a82408baae0ab88e
I20260810 18:50:28.928223 327 raft_consensus.cc:2781] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:50:28.928690 327 raft_consensus.cc:686] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 3 LEADER]: Becoming Leader. State: Replica: 8314391cfbd645b4b780d02a43b4661d, State: Running, Role: LEADER
I20260810 18:50:28.929517 327 consensus_queue.cc:237] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:28.932703 362 ts_tablet_manager.cc:1397] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Time spent bootstrapping tablet: real 0.122s user 0.082s sys 0.022s
I20260810 18:50:28.935371 362 raft_consensus.cc:348] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:28.936142 362 raft_consensus.cc:729] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Initialized, Role: FOLLOWER
I20260810 18:50:28.936650 328 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 3 election: Requested vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:28.936985 362 consensus_queue.cc:260] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:28.947126 362 ts_tablet_manager.cc:1428] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 18:50:28.948192 362 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Bootstrap starting.
I20260810 18:50:28.952034 32560 catalog_manager.cc:5495] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d reported cstate change: term changed from 2 to 3, leader changed from 770b39bfac164d40a82408baae0ab88e (127.28.93.129) to 8314391cfbd645b4b780d02a43b4661d (127.28.93.131). New cstate: current_term: 3 leader_uuid: "8314391cfbd645b4b780d02a43b4661d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: HEALTHY } } }
W20260810 18:50:28.997977 31886 consensus_peers.cc:487] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e -> Peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Couldn't send request to peer 770b39bfac164d40a82408baae0ab88e. Status: Network error: Client connection negotiation failed: client connection to 127.28.93.129:43783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:29.009687 32560 catalog_manager.cc:5495] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e reported cstate change: term changed from 4 to 5, leader changed from 770b39bfac164d40a82408baae0ab88e (127.28.93.129) to f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130). New cstate: current_term: 5 leader_uuid: "f58326f8fbd44e1984c2ce30da077d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:29.014873 32068 raft_consensus.cc:1261] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 5 FOLLOWER]: Refusing update from remote peer f58326f8fbd44e1984c2ce30da077d2e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260810 18:50:29.018373 406 consensus_queue.cc:1035] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:50:29.023499 31937 raft_consensus.cc:1261] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Refusing update from remote peer 8314391cfbd645b4b780d02a43b4661d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260810 18:50:29.025166 327 consensus_queue.cc:1035] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260810 18:50:29.040253 32009 consensus_peers.cc:487] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d -> Peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Couldn't send request to peer 770b39bfac164d40a82408baae0ab88e. Status: Network error: Client connection negotiation failed: client connection to 127.28.93.129:43783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:29.070180 362 tablet.cc:1722] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:29.070998 362 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20260810 18:50:29.072184 362 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=260 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:29.073159 362 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Bootstrap complete.
I20260810 18:50:29.090902 429 raft_consensus.cc:482] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:29.091389 429 raft_consensus.cc:504] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.093693 362 ts_tablet_manager.cc:1397] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Time spent bootstrapping tablet: real 0.146s user 0.079s sys 0.015s
I20260810 18:50:29.096016 362 raft_consensus.cc:348] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:29.096701 362 raft_consensus.cc:729] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Initialized, Role: FOLLOWER
I20260810 18:50:29.097316 362 consensus_queue.cc:260] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:29.093739 429 leader_election.cc:288] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:29.107424 362 ts_tablet_manager.cc:1428] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260810 18:50:29.108238 362 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Bootstrap starting.
W20260810 18:50:29.118544 32009 consensus_peers.cc:487] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d -> Peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Couldn't send request to peer 770b39bfac164d40a82408baae0ab88e. Status: Network error: Client connection negotiation failed: client connection to 127.28.93.129:43783: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260810 18:50:29.167057 31937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f58326f8fbd44e1984c2ce30da077d2e" is_pre_election: true
I20260810 18:50:29.171317 32068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0d266df91e4d60b5289e0683068f80" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:29.172611 348 leader_election.cc:302] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e; no voters: 8314391cfbd645b4b780d02a43b4661d, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:29.176018 429 raft_consensus.cc:2726] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260810 18:50:29.180594 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:43783
I20260810 18:50:29.180680 503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:43783 every 8 connection(s)
I20260810 18:50:29.234866 508 heartbeater.cc:347] Connected to a master server at 127.28.93.188:36699
I20260810 18:50:29.235318 508 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:29.239231 508 heartbeater.cc:511] Master 127.28.93.188:36699 requested a full tablet report, sending...
I20260810 18:50:29.238937 362 tablet.cc:1722] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:29.244887 362 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20260810 18:50:29.245985 362 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=1 applied=5 ignored=0} inserts{seen=276 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:29.246768 362 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Bootstrap complete.
I20260810 18:50:29.248528 32698 ts_manager.cc:194] Registered new tserver with Master: 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:29.255317 362 ts_tablet_manager.cc:1397] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Time spent bootstrapping tablet: real 0.147s user 0.089s sys 0.028s
I20260810 18:50:29.255612 509 heartbeater.cc:347] Connected to a master server at 127.28.93.189:43453
I20260810 18:50:29.256070 509 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:29.256865 509 heartbeater.cc:511] Master 127.28.93.189:43453 requested a full tablet report, sending...
I20260810 18:50:29.257480 511 heartbeater.cc:347] Connected to a master server at 127.28.93.190:38651
I20260810 18:50:29.257359 362 raft_consensus.cc:348] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.257871 511 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:29.258100 362 raft_consensus.cc:729] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Initialized, Role: FOLLOWER
I20260810 18:50:29.258620 511 heartbeater.cc:511] Master 127.28.93.190:38651 requested a full tablet report, sending...
I20260810 18:50:29.258771 362 consensus_queue.cc:260] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.260690 362 ts_tablet_manager.cc:1428] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260810 18:50:29.269656 32560 ts_manager.cc:194] Re-registered known tserver with Master: 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:29.277182 32560 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59556
I20260810 18:50:29.277447 423 raft_consensus.cc:3037] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:50:29.278239 32406 ts_manager.cc:194] Re-registered known tserver with Master: 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783)
I20260810 18:50:29.282855 511 heartbeater.cc:503] Master 127.28.93.190:38651 was elected leader, sending a full tablet report...
I20260810 18:50:29.316246 423 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Advancing to term 5
I20260810 18:50:29.433797 29046 tablet_server.cc:178] TabletServer@127.28.93.130:0 shutting down...
I20260810 18:50:29.471295 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:29.471948 29046 tablet_replica.cc:330] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: stopping tablet replica
I20260810 18:50:29.472484 29046 raft_consensus.cc:2229] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 5 LEADER]: Raft consensus shutting down.
I20260810 18:50:29.473189 29046 raft_consensus.cc:2258] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 5 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:29.475507 29046 tablet_replica.cc:330] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: stopping tablet replica
I20260810 18:50:29.476003 29046 raft_consensus.cc:2229] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:29.476450 29046 raft_consensus.cc:2258] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:29.478749 29046 tablet_replica.cc:330] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: stopping tablet replica
I20260810 18:50:29.479246 29046 raft_consensus.cc:2229] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:29.479624 29046 raft_consensus.cc:2258] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 4 FOLLOWER]: Raft consensus is shut down!
W20260810 18:50:29.479995 32008 consensus_peers.cc:487] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d -> Peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Couldn't send request to peer f58326f8fbd44e1984c2ce30da077d2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:29.500862 29046 tablet_server.cc:194] TabletServer@127.28.93.130:0 shutdown complete.
W20260810 18:50:29.506592 32008 consensus_peers.cc:487] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d -> Peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Couldn't send request to peer f58326f8fbd44e1984c2ce30da077d2e. Status: Network error: Client connection negotiation failed: client connection to 127.28.93.130:45437: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:29.513556 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:29.520119 527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:29.521299 528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:29.521572 531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:29.527994 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:29.531627 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:29.531883 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:29.532092 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387829532075 us; error 0 us; skew 500 ppm
I20260810 18:50:29.532583 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:29.535127 29046 webserver.cc:460] Webserver started at http://127.28.93.130:46783/ using document root <none> and password file <none>
I20260810 18:50:29.535573 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:29.535763 29046 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260810 18:50:29.539218 29046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260810 18:50:29.542304 538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:29.543077 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:50:29.543349 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-1-root
uuid: "f58326f8fbd44e1984c2ce30da077d2e"
format_stamp: "Formatted at 2026-08-10 18:50:22 on dist-test-slave-nj21"
I20260810 18:50:29.543598 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:29.566651 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:29.567699 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:29.568881 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:29.585975 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20260810 18:50:29.586243 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260810 18:50:29.590390 551 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap starting.
I20260810 18:50:29.599439 32205 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:29.600217 29046 ts_tablet_manager.cc:610] Registered 3 tablets
I20260810 18:50:29.599965 32205 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.600484 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.014s user 0.014s sys 0.000s
I20260810 18:50:29.601989 32205 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783), f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:29.603655 423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "8314391cfbd645b4b780d02a43b4661d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "770b39bfac164d40a82408baae0ab88e" is_pre_election: true
W20260810 18:50:29.610801 32008 leader_election.cc:334] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Network error: Client connection negotiation failed: client connection to 127.28.93.130:45437: connect: Connection refused (error 111)
I20260810 18:50:29.611323 32008 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8314391cfbd645b4b780d02a43b4661d; no voters: 770b39bfac164d40a82408baae0ab88e, f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:29.612377 32205 raft_consensus.cc:2726] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260810 18:50:29.615918 429 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:29.616436 429 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.618420 429 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:29.620949 32068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d" is_pre_election: true
I20260810 18:50:29.621590 32068 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770b39bfac164d40a82408baae0ab88e in term 5.
I20260810 18:50:29.622617 348 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d; no voters:
I20260810 18:50:29.623400 429 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260810 18:50:29.623818 429 raft_consensus.cc:482] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 5 FOLLOWER]: Starting leader election (detected failure of leader f58326f8fbd44e1984c2ce30da077d2e)
I20260810 18:50:29.624197 429 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:50:29.624451 347 leader_election.cc:334] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Network error: Client connection negotiation failed: client connection to 127.28.93.130:45437: connect: Connection refused (error 111)
I20260810 18:50:29.631472 429 raft_consensus.cc:504] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.633997 429 leader_election.cc:288] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 6 election: Requested vote from peers f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437), 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737)
I20260810 18:50:29.634989 32068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f88e634539c49fd95f67aaedf60a5b5" candidate_uuid: "770b39bfac164d40a82408baae0ab88e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "8314391cfbd645b4b780d02a43b4661d"
I20260810 18:50:29.635632 32068 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 5 FOLLOWER]: Advancing to term 6
W20260810 18:50:29.639037 347 leader_election.cc:334] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Network error: Client connection negotiation failed: client connection to 127.28.93.130:45437: connect: Connection refused (error 111)
I20260810 18:50:29.642711 32068 raft_consensus.cc:2454] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770b39bfac164d40a82408baae0ab88e in term 6.
I20260810 18:50:29.643862 348 leader_election.cc:302] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 770b39bfac164d40a82408baae0ab88e, 8314391cfbd645b4b780d02a43b4661d; no voters: f58326f8fbd44e1984c2ce30da077d2e
I20260810 18:50:29.644738 429 raft_consensus.cc:2781] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 6 FOLLOWER]: Leader election won for term 6
I20260810 18:50:29.645884 429 raft_consensus.cc:686] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 6 LEADER]: Becoming Leader. State: Replica: 770b39bfac164d40a82408baae0ab88e, State: Running, Role: LEADER
I20260810 18:50:29.646801 429 consensus_queue.cc:237] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.660178 32560 catalog_manager.cc:5495] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e reported cstate change: term changed from 5 to 6, leader changed from f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130) to 770b39bfac164d40a82408baae0ab88e (127.28.93.129). New cstate: current_term: 6 leader_uuid: "770b39bfac164d40a82408baae0ab88e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:29.679250 551 tablet.cc:1722] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:29.680027 551 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20260810 18:50:29.682562 551 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=266 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:29.683400 551 tablet_bootstrap.cc:492] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap complete.
I20260810 18:50:29.707129 551 ts_tablet_manager.cc:1397] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Time spent bootstrapping tablet: real 0.117s user 0.086s sys 0.020s
I20260810 18:50:29.709398 551 raft_consensus.cc:348] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.710105 551 raft_consensus.cc:729] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Initialized, Role: FOLLOWER
I20260810 18:50:29.710678 551 consensus_queue.cc:260] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.721380 551 ts_tablet_manager.cc:1428] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260810 18:50:29.722141 551 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap starting.
W20260810 18:50:29.729727 347 consensus_peers.cc:487] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e -> Peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Couldn't send request to peer f58326f8fbd44e1984c2ce30da077d2e. Status: Network error: Client connection negotiation failed: client connection to 127.28.93.130:45437: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:29.764318 32068 raft_consensus.cc:1261] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 6 FOLLOWER]: Refusing update from remote peer 770b39bfac164d40a82408baae0ab88e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260810 18:50:29.765914 429 consensus_queue.cc:1035] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:50:29.793926 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.130:45437
I20260810 18:50:29.794014 660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.130:45437 every 8 connection(s)
I20260810 18:50:29.799273 551 tablet.cc:1722] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:29.800020 551 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20260810 18:50:29.801007 551 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=1 applied=4 ignored=0} inserts{seen=260 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:29.801769 551 tablet_bootstrap.cc:492] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap complete.
I20260810 18:50:29.818574 551 ts_tablet_manager.cc:1397] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Time spent bootstrapping tablet: real 0.097s user 0.075s sys 0.007s
E20260810 18:50:29.822674 551 ts_tablet_manager.cc:1439] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Tablet failed to start: Corruption: Unable to start RaftConsensus: The last op in the WAL with id 4.4 has a term (4) that is greater than the latest recorded term, which is 3
I20260810 18:50:29.822973 551 ts_tablet_manager.cc:1428] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 18:50:29.823354 551 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:29.823613 551 raft_consensus.cc:2229] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:29.824039 551 raft_consensus.cc:2258] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:29.835028 551 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap starting.
I20260810 18:50:29.849957 663 heartbeater.cc:347] Connected to a master server at 127.28.93.188:36699
I20260810 18:50:29.850344 663 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:29.851274 663 heartbeater.cc:511] Master 127.28.93.188:36699 requested a full tablet report, sending...
I20260810 18:50:29.855269 32698 ts_manager.cc:194] Re-registered known tserver with Master: f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:29.860185 666 heartbeater.cc:347] Connected to a master server at 127.28.93.189:43453
I20260810 18:50:29.860539 666 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:29.861230 666 heartbeater.cc:511] Master 127.28.93.189:43453 requested a full tablet report, sending...
I20260810 18:50:29.865700 32406 ts_manager.cc:194] Re-registered known tserver with Master: f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
I20260810 18:50:29.874138 667 heartbeater.cc:347] Connected to a master server at 127.28.93.190:38651
I20260810 18:50:29.874475 667 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:29.875224 667 heartbeater.cc:511] Master 127.28.93.190:38651 requested a full tablet report, sending...
I20260810 18:50:29.878075 32562 ts_manager.cc:194] Re-registered known tserver with Master: f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437)
W20260810 18:50:29.879802 32562 catalog_manager.cc:5408] Tablet 618c53eed4a94a9eab8d7c30da86b31f (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9]) has failed on TS f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Corruption: Unable to start RaftConsensus: The last op in the WAL with id 4.4 has a term (4) that is greater than the latest recorded term, which is 3
I20260810 18:50:29.880251 32562 catalog_manager.cc:5587] TS f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437) does not have the latest schema for tablet 0f88e634539c49fd95f67aaedf60a5b5 (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9]). Expected version 1 got 0
I20260810 18:50:29.884352 32562 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59566
I20260810 18:50:29.889575 667 heartbeater.cc:503] Master 127.28.93.190:38651 was elected leader, sending a full tablet report...
W20260810 18:50:29.895020 32560 catalog_manager.cc:5408] Tablet 618c53eed4a94a9eab8d7c30da86b31f (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9]) has failed on TS f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Corruption: Unable to start RaftConsensus: The last op in the WAL with id 4.4 has a term (4) that is greater than the latest recorded term, which is 3
I20260810 18:50:29.894965 32008 consensus_queue.cc:579] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: The tablet replica hosted on peer f58326f8fbd44e1984c2ce30da077d2e has failed
I20260810 18:50:29.902220 32560 catalog_manager.cc:5587] TS f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437) does not have the latest schema for tablet 0f88e634539c49fd95f67aaedf60a5b5 (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9]). Expected version 1 got 0
I20260810 18:50:29.910467 551 tablet.cc:1722] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260810 18:50:29.911206 551 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20260810 18:50:29.913753 551 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=276 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:29.914580 551 tablet_bootstrap.cc:492] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Bootstrap complete.
I20260810 18:50:29.921175 551 ts_tablet_manager.cc:1397] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Time spent bootstrapping tablet: real 0.086s user 0.067s sys 0.008s
I20260810 18:50:29.923092 551 raft_consensus.cc:348] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.923605 551 raft_consensus.cc:729] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58326f8fbd44e1984c2ce30da077d2e, State: Initialized, Role: FOLLOWER
I20260810 18:50:29.924252 551 consensus_queue.cc:260] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 45437 } } peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } }
I20260810 18:50:29.926019 551 ts_tablet_manager.cc:1428] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:50:29.934911 32068 consensus_queue.cc:237] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
I20260810 18:50:29.939463 423 raft_consensus.cc:1261] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Refusing update from remote peer 8314391cfbd645b4b780d02a43b4661d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260810 18:50:29.940955 328 consensus_queue.cc:1035] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260810 18:50:29.946106 32205 raft_consensus.cc:2932] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index -1 to 5, VOTER f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } }
I20260810 18:50:29.947574 423 raft_consensus.cc:2932] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, VOTER f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } }
I20260810 18:50:29.954149 32531 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 618c53eed4a94a9eab8d7c30da86b31f with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:50:29.960531 612 raft_consensus.cc:3037] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 5 FOLLOWER]: Advancing to term 6
I20260810 18:50:29.959930 32562 catalog_manager.cc:5495] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d reported cstate change: config changed from index -1 to 5, VOTER f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130) evicted. New cstate: current_term: 4 leader_uuid: "8314391cfbd645b4b780d02a43b4661d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:29.989143 32068 consensus_queue.cc:237] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: NON_VOTER last_known_addr { host: "127.28.93.130" port: 45437 } attrs { promote: true } }
I20260810 18:50:29.994711 32008 consensus_queue.cc:579] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: The tablet replica hosted on peer f58326f8fbd44e1984c2ce30da077d2e has failed
W20260810 18:50:29.995121 32008 consensus_peers.cc:487] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d -> Peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Couldn't send request to peer f58326f8fbd44e1984c2ce30da077d2e. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Corruption: Unable to start RaftConsensus: The last op in the WAL with id 4.4 has a term (4) that is greater than the latest recorded term, which is 3. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:29.994732 423 raft_consensus.cc:1261] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Refusing update from remote peer 8314391cfbd645b4b780d02a43b4661d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260810 18:50:29.996876 328 consensus_queue.cc:1035] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Connected to new peer: Peer: permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:50:30.000753 594 tablet_service.cc:1514] Processing DeleteTablet for tablet 618c53eed4a94a9eab8d7c30da86b31f with delete_type TABLET_DATA_TOMBSTONED (TS f58326f8fbd44e1984c2ce30da077d2e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42850
I20260810 18:50:30.004710 692 ts_tablet_manager.cc:1905] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:50:30.004315 32205 raft_consensus.cc:2932] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, NON_VOTER f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: NON_VOTER last_known_addr { host: "127.28.93.130" port: 45437 } attrs { promote: true } } }
I20260810 18:50:30.005975 423 raft_consensus.cc:2932] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, NON_VOTER f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: NON_VOTER last_known_addr { host: "127.28.93.130" port: 45437 } attrs { promote: true } } }
I20260810 18:50:30.011289 32531 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 618c53eed4a94a9eab8d7c30da86b31f with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260810 18:50:30.019271 692 ts_tablet_manager.cc:1918] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.4
I20260810 18:50:30.021632 692 log.cc:1196] T 618c53eed4a94a9eab8d7c30da86b31f P f58326f8fbd44e1984c2ce30da077d2e: Deleting WAL directory at /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/minicluster-data/ts-1-root/wals/618c53eed4a94a9eab8d7c30da86b31f
I20260810 18:50:30.023438 32530 catalog_manager.cc:4841] TS f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): tablet 618c53eed4a94a9eab8d7c30da86b31f (table NewTableName [id=c384a89a9c1b42f8ae3b3f1d6815aba9]) successfully deleted
I20260810 18:50:30.025089 32562 catalog_manager.cc:5495] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d reported cstate change: config changed from index 5 to 6, NON_VOTER f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130) added. New cstate: current_term: 4 leader_uuid: "8314391cfbd645b4b780d02a43b4661d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: NON_VOTER last_known_addr { host: "127.28.93.130" port: 45437 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } }
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:906: Failure
Failed
Bad status: Corruption: Unable to start RaftConsensus: The last op in the WAL with id 4.4 has a term (4) that is greater than the latest recorded term, which is 3
I20260810 18:50:30.032630 29046 tablet_server.cc:178] TabletServer@127.28.93.129:43783 shutting down...
I20260810 18:50:30.057688 32068 consensus_queue.cc:237] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } }
W20260810 18:50:30.069715 32009 consensus_peers.cc:487] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d -> Peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Couldn't send request to peer 770b39bfac164d40a82408baae0ab88e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:30.079869 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:30.080616 29046 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:30.081275 29046 raft_consensus.cc:2229] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 6 LEADER]: Raft consensus shutting down.
I20260810 18:50:30.082188 29046 raft_consensus.cc:2258] T 0f88e634539c49fd95f67aaedf60a5b5 P 770b39bfac164d40a82408baae0ab88e [term 6 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:30.084537 29046 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:30.084985 29046 raft_consensus.cc:2229] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:30.085361 29046 raft_consensus.cc:2258] T 618c53eed4a94a9eab8d7c30da86b31f P 770b39bfac164d40a82408baae0ab88e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:30.087487 29046 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:30.087939 29046 raft_consensus.cc:2229] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:30.088285 29046 raft_consensus.cc:2258] T 5f0d266df91e4d60b5289e0683068f80 P 770b39bfac164d40a82408baae0ab88e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:30.100898 29046 tablet_server.cc:194] TabletServer@127.28.93.129:43783 shutdown complete.
W20260810 18:50:30.104276 32009 consensus_peers.cc:487] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d -> Peer 770b39bfac164d40a82408baae0ab88e (127.28.93.129:43783): Couldn't send request to peer 770b39bfac164d40a82408baae0ab88e. Status: Network error: Client connection negotiation failed: client connection to 127.28.93.129:43783: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:30.111306 29046 tablet_server.cc:178] TabletServer@127.28.93.130:45437 shutting down...
W20260810 18:50:30.128747 32008 consensus_peers.cc:487] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d -> Peer f58326f8fbd44e1984c2ce30da077d2e (127.28.93.130:45437): Couldn't send request to peer f58326f8fbd44e1984c2ce30da077d2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:30.133252 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:30.134120 29046 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:30.134672 29046 raft_consensus.cc:2229] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 6 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:30.135042 29046 raft_consensus.cc:2258] T 0f88e634539c49fd95f67aaedf60a5b5 P f58326f8fbd44e1984c2ce30da077d2e [term 6 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:30.137303 29046 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:30.137729 29046 raft_consensus.cc:2229] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:30.138023 29046 raft_consensus.cc:2258] T 5f0d266df91e4d60b5289e0683068f80 P f58326f8fbd44e1984c2ce30da077d2e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:30.159042 29046 tablet_server.cc:194] TabletServer@127.28.93.130:45437 shutdown complete.
I20260810 18:50:30.169699 29046 tablet_server.cc:178] TabletServer@127.28.93.131:0 shutting down...
I20260810 18:50:30.198674 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:30.199296 29046 tablet_replica.cc:330] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d: stopping tablet replica
I20260810 18:50:30.199847 29046 raft_consensus.cc:2229] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 3 LEADER]: Raft consensus shutting down.
I20260810 18:50:30.200553 29046 raft_consensus.cc:2258] T 5f0d266df91e4d60b5289e0683068f80 P 8314391cfbd645b4b780d02a43b4661d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:30.202569 29046 tablet_replica.cc:330] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d: stopping tablet replica
I20260810 18:50:30.203039 29046 raft_consensus.cc:2229] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 4 LEADER]: Raft consensus shutting down.
I20260810 18:50:30.203565 29046 pending_rounds.cc:62] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d: Trying to abort 1 pending ops.
I20260810 18:50:30.203774 29046 pending_rounds.cc:68] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d: Aborting op as it isn't in flight: id { term: 4 index: 7 } timestamp: 7317044551911157760 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "618c53eed4a94a9eab8d7c30da86b31f" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } peers { permanent_uuid: "f58326f8fbd44e1984c2ce30da077d2e" member_type: NON_VOTER last_known_addr { host: "127.28.93.130" port: 45437 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8314391cfbd645b4b780d02a43b4661d" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 37737 } } peers { permanent_uuid: "770b39bfac164d40a82408baae0ab88e" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43783 } } } }
I20260810 18:50:30.204427 29046 raft_consensus.cc:2894] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 4 LEADER]: Aborting config change with OpId 4.7: Aborted: Op aborted
I20260810 18:50:30.204677 29046 raft_consensus.cc:2866] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 4 LEADER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260810 18:50:30.205189 29046 raft_consensus.cc:2258] T 618c53eed4a94a9eab8d7c30da86b31f P 8314391cfbd645b4b780d02a43b4661d [term 4 FOLLOWER]: Raft consensus is shut down!
W20260810 18:50:30.205585 32531 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 618c53eed4a94a9eab8d7c30da86b31f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737) due to error UNKNOWN_ERROR; will retry: Aborted: Op aborted
I20260810 18:50:30.207419 29046 tablet_replica.cc:330] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d: stopping tablet replica
I20260810 18:50:30.207911 29046 raft_consensus.cc:2229] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 6 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:30.208271 29046 raft_consensus.cc:2258] T 0f88e634539c49fd95f67aaedf60a5b5 P 8314391cfbd645b4b780d02a43b4661d [term 6 FOLLOWER]: Raft consensus is shut down!
W20260810 18:50:30.228785 29046 rolling_log.cc:182] Unable to compress old log file: Not found: Unable to open input file to compress: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0/ts_tablet_manager-itest.dist-test-slave-nj21.slave.diagnostics.20260810-185023.1.29046: No such file or directory (error 2)
I20260810 18:50:30.229367 29046 tablet_server.cc:194] TabletServer@127.28.93.131:0 shutdown complete.
I20260810 18:50:30.239868 29046 master.cc:537] Master@127.28.93.190:38651 shutting down...
W20260810 18:50:30.259111 32531 catalog_manager.cc:4584] TS 8314391cfbd645b4b780d02a43b4661d (127.28.93.131:37737): ChangeConfig:REMOVE_PEER RPC failed for tablet 618c53eed4a94a9eab8d7c30da86b31f: Network error: Client connection negotiation failed: client connection to 127.28.93.131:37737: connect: Connection refused (error 111)
I20260810 18:50:30.325464 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 4 LEADER]: Raft consensus shutting down.
I20260810 18:50:30.326301 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f7b3db47356949cc9049320effc90535 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:30.326609 29046 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:30.337404 29046 master.cc:552] Master@127.28.93.190:38651 shutdown complete.
I20260810 18:50:30.348223 29046 master.cc:537] Master@127.28.93.189:43453 shutting down...
I20260810 18:50:30.362970 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:30.363432 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 79547306558a49778d11f86a5276bb76 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:30.363756 29046 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:30.382299 29046 master.cc:552] Master@127.28.93.189:43453 shutdown complete.
I20260810 18:50:30.392429 29046 master.cc:537] Master@127.28.93.188:36699 shutting down...
I20260810 18:50:30.405802 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:30.406230 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bc51947e5d9c444e85b6c335c9cf79d8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:30.406483 29046 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:30.424469 29046 master.cc:552] Master@127.28.93.188:36699 shutdown complete.
I20260810 18:50:30.440769 29046 test_util.cc:175] -----------------------------------------------
I20260810 18:50:30.440940 29046 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786387803638833-29046-0
[ FAILED ] TsTabletManagerITest.TestTableStats (8362 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20260810 18:50:30.446595 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:40151
I20260810 18:50:30.447532 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:30.451965 704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:30.452847 705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:30.454974 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:30.455056 710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:30.455996 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:30.456171 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:30.456328 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387830456314 us; error 0 us; skew 500 ppm
I20260810 18:50:30.456838 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:30.458945 29046 webserver.cc:460] Webserver started at http://127.28.93.190:44229/ using document root <none> and password file <none>
I20260810 18:50:30.459376 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:30.459537 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:30.459810 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:30.460848 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "2bef2addf1a54743bec9bcd3c02215c7"
format_stamp: "Formatted at 2026-08-10 18:50:30 on dist-test-slave-nj21"
I20260810 18:50:30.464952 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:50:30.467945 722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:30.468585 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:30.468842 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "2bef2addf1a54743bec9bcd3c02215c7"
format_stamp: "Formatted at 2026-08-10 18:50:30 on dist-test-slave-nj21"
I20260810 18:50:30.469092 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:30.478142 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:30.479046 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:30.513161 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:40151
I20260810 18:50:30.513247 795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:40151 every 8 connection(s)
I20260810 18:50:30.516722 796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:30.525864 796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7: Bootstrap starting.
I20260810 18:50:30.529723 796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:30.533437 796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7: No bootstrap required, opened a new log
I20260810 18:50:30.535248 796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef2addf1a54743bec9bcd3c02215c7" member_type: VOTER }
I20260810 18:50:30.535610 796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:30.535866 796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bef2addf1a54743bec9bcd3c02215c7, State: Initialized, Role: FOLLOWER
I20260810 18:50:30.536332 796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef2addf1a54743bec9bcd3c02215c7" member_type: VOTER }
I20260810 18:50:30.536785 796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:30.537061 796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:30.537297 796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:30.541533 796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef2addf1a54743bec9bcd3c02215c7" member_type: VOTER }
I20260810 18:50:30.542003 796 leader_election.cc:302] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bef2addf1a54743bec9bcd3c02215c7; no voters:
I20260810 18:50:30.543063 796 leader_election.cc:288] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:30.543329 800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:30.544791 800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 1 LEADER]: Becoming Leader. State: Replica: 2bef2addf1a54743bec9bcd3c02215c7, State: Running, Role: LEADER
I20260810 18:50:30.545446 800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef2addf1a54743bec9bcd3c02215c7" member_type: VOTER }
I20260810 18:50:30.546074 796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:30.549160 802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bef2addf1a54743bec9bcd3c02215c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef2addf1a54743bec9bcd3c02215c7" member_type: VOTER } }
I20260810 18:50:30.549938 802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:30.549322 803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bef2addf1a54743bec9bcd3c02215c7. Latest consensus state: current_term: 1 leader_uuid: "2bef2addf1a54743bec9bcd3c02215c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef2addf1a54743bec9bcd3c02215c7" member_type: VOTER } }
I20260810 18:50:30.551196 803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:30.551636 809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:30.556102 809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:30.561609 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:30.564260 809 catalog_manager.cc:1347] Generated new cluster ID: 2c1026dff1b244f4975092fcf56c7f74
I20260810 18:50:30.564463 809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:30.578722 809 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:30.580317 809 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:30.589720 809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7: Generated new TSK 0
I20260810 18:50:30.590297 809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:30.597325 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:30.602890 835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:30.603395 836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:30.605636 840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:30.606453 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:30.607187 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:30.607376 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:30.607520 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387830607505 us; error 0 us; skew 500 ppm
I20260810 18:50:30.608023 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:30.610121 29046 webserver.cc:460] Webserver started at http://127.28.93.129:34819/ using document root <none> and password file <none>
I20260810 18:50:30.610563 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:30.610734 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:30.610970 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:30.612033 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "4b5e867ac5d04ec3a8e336307db60b05"
format_stamp: "Formatted at 2026-08-10 18:50:30 on dist-test-slave-nj21"
I20260810 18:50:30.616295 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:50:30.619200 851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:30.619892 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:30.620146 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "4b5e867ac5d04ec3a8e336307db60b05"
format_stamp: "Formatted at 2026-08-10 18:50:30 on dist-test-slave-nj21"
I20260810 18:50:30.620400 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:30.645821 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:30.646773 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:30.648283 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:30.650424 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:30.650599 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:30.650806 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:30.650945 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:30.692682 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:41211
I20260810 18:50:30.692781 955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:41211 every 8 connection(s)
I20260810 18:50:30.698282 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:30.706099 962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:30.706888 963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:30.712157 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:30.713253 957 heartbeater.cc:347] Connected to a master server at 127.28.93.190:40151
I20260810 18:50:30.713598 957 heartbeater.cc:464] Registering TS with master...
W20260810 18:50:30.713723 965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:30.714337 957 heartbeater.cc:511] Master 127.28.93.190:40151 requested a full tablet report, sending...
I20260810 18:50:30.714710 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:30.714973 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:30.715169 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387830715151 us; error 0 us; skew 500 ppm
I20260810 18:50:30.715873 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:30.716436 751 ts_manager.cc:194] Registered new tserver with Master: 4b5e867ac5d04ec3a8e336307db60b05 (127.28.93.129:41211)
I20260810 18:50:30.718191 751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42994
I20260810 18:50:30.720643 29046 webserver.cc:460] Webserver started at http://127.28.93.130:40051/ using document root <none> and password file <none>
I20260810 18:50:30.721139 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:30.721330 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:30.721608 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:30.722648 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-1-root/instance:
uuid: "28897e9be92e4a4a8ce102f6421f89bb"
format_stamp: "Formatted at 2026-08-10 18:50:30 on dist-test-slave-nj21"
I20260810 18:50:30.726754 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:50:30.729728 971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:30.730407 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:50:30.730650 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-1-root
uuid: "28897e9be92e4a4a8ce102f6421f89bb"
format_stamp: "Formatted at 2026-08-10 18:50:30 on dist-test-slave-nj21"
I20260810 18:50:30.730904 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:30.770401 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:30.771385 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:30.772782 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:30.774899 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:30.775069 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:30.775291 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:30.775422 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:30.811980 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.130:34549
I20260810 18:50:30.812076 1082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.130:34549 every 8 connection(s)
I20260810 18:50:30.816278 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:30.822923 1087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:30.823722 1088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:30.827401 1090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:30.829746 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:30.830621 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:30.830830 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:30.831009 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387830830992 us; error 0 us; skew 500 ppm
I20260810 18:50:30.831555 1083 heartbeater.cc:347] Connected to a master server at 127.28.93.190:40151
I20260810 18:50:30.831617 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:30.831969 1083 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:30.832677 1083 heartbeater.cc:511] Master 127.28.93.190:40151 requested a full tablet report, sending...
I20260810 18:50:30.834411 29046 webserver.cc:460] Webserver started at http://127.28.93.131:46717/ using document root <none> and password file <none>
I20260810 18:50:30.834759 751 ts_manager.cc:194] Registered new tserver with Master: 28897e9be92e4a4a8ce102f6421f89bb (127.28.93.130:34549)
I20260810 18:50:30.835052 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:30.835299 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:30.835597 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:30.836910 751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43004
I20260810 18:50:30.836995 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-2-root/instance:
uuid: "ec9fa5758ce94c5fbd95725a197bcb74"
format_stamp: "Formatted at 2026-08-10 18:50:30 on dist-test-slave-nj21"
I20260810 18:50:30.842052 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:50:30.845111 1096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:30.845861 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:50:30.846104 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-2-root
uuid: "ec9fa5758ce94c5fbd95725a197bcb74"
format_stamp: "Formatted at 2026-08-10 18:50:30 on dist-test-slave-nj21"
I20260810 18:50:30.846331 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:30.858978 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:30.859957 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:30.861243 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:30.863339 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:30.863509 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:30.863749 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:30.863883 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:30.899837 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.131:41361
I20260810 18:50:30.899933 1196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.131:41361 every 8 connection(s)
I20260810 18:50:30.913621 1197 heartbeater.cc:347] Connected to a master server at 127.28.93.190:40151
I20260810 18:50:30.913938 1197 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:30.914551 1197 heartbeater.cc:511] Master 127.28.93.190:40151 requested a full tablet report, sending...
I20260810 18:50:30.916234 751 ts_manager.cc:194] Registered new tserver with Master: ec9fa5758ce94c5fbd95725a197bcb74 (127.28.93.131:41361)
I20260810 18:50:30.917070 29046 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01432277s
I20260810 18:50:30.917683 751 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43016
I20260810 18:50:30.950703 751 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260810 18:50:30.952147 751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:50:30.991086 1020 tablet_service.cc:1467] Processing CreateTablet for tablet fdf1286333c44421a23476add54d7e01 (DEFAULT_TABLE table=test-table [id=992d2c2f851e4d009812afa279c4d445]), partition=
I20260810 18:50:30.992398 1020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf1286333c44421a23476add54d7e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:30.995760 1137 tablet_service.cc:1467] Processing CreateTablet for tablet fdf1286333c44421a23476add54d7e01 (DEFAULT_TABLE table=test-table [id=992d2c2f851e4d009812afa279c4d445]), partition=
I20260810 18:50:30.996623 1137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf1286333c44421a23476add54d7e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:31.004875 1225 tablet_bootstrap.cc:492] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: Bootstrap starting.
I20260810 18:50:31.005198 899 tablet_service.cc:1467] Processing CreateTablet for tablet fdf1286333c44421a23476add54d7e01 (DEFAULT_TABLE table=test-table [id=992d2c2f851e4d009812afa279c4d445]), partition=
I20260810 18:50:31.006309 899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf1286333c44421a23476add54d7e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:31.009105 1226 tablet_bootstrap.cc:492] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Bootstrap starting.
I20260810 18:50:31.010749 1225 tablet_bootstrap.cc:654] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:31.014648 1226 tablet_bootstrap.cc:654] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:31.015935 1225 tablet_bootstrap.cc:492] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: No bootstrap required, opened a new log
I20260810 18:50:31.016372 1225 ts_tablet_manager.cc:1397] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260810 18:50:31.019040 1225 raft_consensus.cc:348] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.019826 1225 raft_consensus.cc:374] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:31.020135 1225 raft_consensus.cc:729] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28897e9be92e4a4a8ce102f6421f89bb, State: Initialized, Role: FOLLOWER
I20260810 18:50:31.020892 1225 consensus_queue.cc:260] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.022605 1231 tablet_bootstrap.cc:492] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05: Bootstrap starting.
I20260810 18:50:31.022753 1226 tablet_bootstrap.cc:492] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: No bootstrap required, opened a new log
I20260810 18:50:31.023201 1083 heartbeater.cc:503] Master 127.28.93.190:40151 was elected leader, sending a full tablet report...
I20260810 18:50:31.023483 1225 ts_tablet_manager.cc:1428] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260810 18:50:31.023789 1225 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260810 18:50:31.024300 1226 ts_tablet_manager.cc:1397] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260810 18:50:31.026623 1226 raft_consensus.cc:348] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.027221 1226 raft_consensus.cc:374] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:31.027441 1226 raft_consensus.cc:729] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9fa5758ce94c5fbd95725a197bcb74, State: Initialized, Role: FOLLOWER
I20260810 18:50:31.028134 1231 tablet_bootstrap.cc:654] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:31.028004 1226 consensus_queue.cc:260] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.030131 1197 heartbeater.cc:503] Master 127.28.93.190:40151 was elected leader, sending a full tablet report...
I20260810 18:50:31.030376 1226 ts_tablet_manager.cc:1428] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
W20260810 18:50:31.030637 1226 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260810 18:50:31.034708 1231 tablet_bootstrap.cc:492] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05: No bootstrap required, opened a new log
I20260810 18:50:31.035144 1231 ts_tablet_manager.cc:1397] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20260810 18:50:31.037022 1231 raft_consensus.cc:348] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.037496 1231 raft_consensus.cc:374] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:31.037729 1231 raft_consensus.cc:729] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5e867ac5d04ec3a8e336307db60b05, State: Initialized, Role: FOLLOWER
I20260810 18:50:31.038249 1231 consensus_queue.cc:260] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.039870 957 heartbeater.cc:503] Master 127.28.93.190:40151 was elected leader, sending a full tablet report...
I20260810 18:50:31.040192 1231 ts_tablet_manager.cc:1428] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
W20260810 18:50:31.040453 1231 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260810 18:50:31.057582 1233 raft_consensus.cc:482] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:31.057955 1233 raft_consensus.cc:504] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.059630 1233 leader_election.cc:288] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec9fa5758ce94c5fbd95725a197bcb74 (127.28.93.131:41361), 4b5e867ac5d04ec3a8e336307db60b05 (127.28.93.129:41211)
I20260810 18:50:31.072469 1153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf1286333c44421a23476add54d7e01" candidate_uuid: "28897e9be92e4a4a8ce102f6421f89bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" is_pre_election: true
I20260810 18:50:31.073028 1153 raft_consensus.cc:2454] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28897e9be92e4a4a8ce102f6421f89bb in term 0.
I20260810 18:50:31.074163 973 leader_election.cc:302] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28897e9be92e4a4a8ce102f6421f89bb, ec9fa5758ce94c5fbd95725a197bcb74; no voters:
I20260810 18:50:31.074185 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf1286333c44421a23476add54d7e01" candidate_uuid: "28897e9be92e4a4a8ce102f6421f89bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e867ac5d04ec3a8e336307db60b05" is_pre_election: true
I20260810 18:50:31.074941 914 raft_consensus.cc:2454] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28897e9be92e4a4a8ce102f6421f89bb in term 0.
I20260810 18:50:31.075084 1233 raft_consensus.cc:2781] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:31.075443 1233 raft_consensus.cc:482] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:31.075846 1233 raft_consensus.cc:3037] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:31.080005 1233 raft_consensus.cc:504] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.081554 1233 leader_election.cc:288] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [CANDIDATE]: Term 1 election: Requested vote from peers ec9fa5758ce94c5fbd95725a197bcb74 (127.28.93.131:41361), 4b5e867ac5d04ec3a8e336307db60b05 (127.28.93.129:41211)
I20260810 18:50:31.083032 1153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf1286333c44421a23476add54d7e01" candidate_uuid: "28897e9be92e4a4a8ce102f6421f89bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec9fa5758ce94c5fbd95725a197bcb74"
I20260810 18:50:31.083472 914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf1286333c44421a23476add54d7e01" candidate_uuid: "28897e9be92e4a4a8ce102f6421f89bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b5e867ac5d04ec3a8e336307db60b05"
I20260810 18:50:31.083606 1153 raft_consensus.cc:3037] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:31.084035 914 raft_consensus.cc:3037] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:31.088721 914 raft_consensus.cc:2454] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28897e9be92e4a4a8ce102f6421f89bb in term 1.
I20260810 18:50:31.089311 1153 raft_consensus.cc:2454] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28897e9be92e4a4a8ce102f6421f89bb in term 1.
I20260810 18:50:31.089474 973 leader_election.cc:302] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28897e9be92e4a4a8ce102f6421f89bb, 4b5e867ac5d04ec3a8e336307db60b05; no voters:
I20260810 18:50:31.090147 1233 raft_consensus.cc:2781] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:31.090988 1233 raft_consensus.cc:686] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 1 LEADER]: Becoming Leader. State: Replica: 28897e9be92e4a4a8ce102f6421f89bb, State: Running, Role: LEADER
I20260810 18:50:31.091820 1233 consensus_queue.cc:237] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.098690 751 catalog_manager.cc:5495] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 28897e9be92e4a4a8ce102f6421f89bb (127.28.93.130). New cstate: current_term: 1 leader_uuid: "28897e9be92e4a4a8ce102f6421f89bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:31.147997 1137 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf1286333c44421a23476add54d7e01 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:49986
I20260810 18:50:31.149276 1244 tablet_replica.cc:330] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: stopping tablet replica
I20260810 18:50:31.149865 1244 raft_consensus.cc:2229] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:31.150252 1244 raft_consensus.cc:2258] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:31.152115 1244 ts_tablet_manager.cc:1905] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:50:31.161147 1244 ts_tablet_manager.cc:1918] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20260810 18:50:31.161453 1244 log.cc:1196] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Deleting WAL directory at /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-2-root/wals/fdf1286333c44421a23476add54d7e01
I20260810 18:50:31.165241 1245 ts_tablet_manager.cc:927] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Initiating tablet copy from peer 28897e9be92e4a4a8ce102f6421f89bb (127.28.93.130:34549)
I20260810 18:50:31.166973 1245 tablet_copy_client.cc:316] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: tablet copy: Beginning tablet copy session from remote peer at address 127.28.93.130:34549
I20260810 18:50:31.176158 1060 tablet_copy_service.cc:140] P 28897e9be92e4a4a8ce102f6421f89bb: Received BeginTabletCopySession request for tablet fdf1286333c44421a23476add54d7e01 from peer ec9fa5758ce94c5fbd95725a197bcb74 ({username='slave'} at 127.0.0.1:42656)
I20260810 18:50:31.176678 1060 tablet_copy_service.cc:161] P 28897e9be92e4a4a8ce102f6421f89bb: Beginning new tablet copy session on tablet fdf1286333c44421a23476add54d7e01 from peer ec9fa5758ce94c5fbd95725a197bcb74 at {username='slave'} at 127.0.0.1:42656: session id = ec9fa5758ce94c5fbd95725a197bcb74-fdf1286333c44421a23476add54d7e01
I20260810 18:50:31.181879 1060 tablet_copy_source_session.cc:205] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:50:31.184201 1245 ts_tablet_manager.cc:1905] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 18:50:31.193070 1245 ts_tablet_manager.cc:1918] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20260810 18:50:31.193609 1245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf1286333c44421a23476add54d7e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:31.198324 1245 tablet_copy_client.cc:791] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: tablet copy: Starting download of 0 data blocks...
I20260810 18:50:31.198767 1245 tablet_copy_client.cc:655] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: tablet copy: Starting download of 1 WAL segments...
I20260810 18:50:31.200362 751 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43038:
table { table_id: "992d2c2f851e4d009812afa279c4d445" table_name: "test-table" }
I20260810 18:50:31.200989 751 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43038:
table { table_id: "992d2c2f851e4d009812afa279c4d445" table_name: "test-table" }
I20260810 18:50:31.202586 1245 tablet_copy_client.cc:531] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:50:31.208441 1245 tablet_bootstrap.cc:492] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Bootstrap starting.
I20260810 18:50:31.210500 751 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7: Sending DeleteTablet for 3 replicas of tablet fdf1286333c44421a23476add54d7e01
I20260810 18:50:31.212313 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf1286333c44421a23476add54d7e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:50:31 UTC) from {username='slave'} at 127.0.0.1:42624
I20260810 18:50:31.213299 1137 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf1286333c44421a23476add54d7e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:50:31 UTC) from {username='slave'} at 127.0.0.1:49956
I20260810 18:50:31.214673 899 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf1286333c44421a23476add54d7e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:50:31 UTC) from {username='slave'} at 127.0.0.1:57018
I20260810 18:50:31.214875 1247 tablet_replica.cc:330] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: stopping tablet replica
W20260810 18:50:31.215839 727 catalog_manager.cc:4821] TS ec9fa5758ce94c5fbd95725a197bcb74 (127.28.93.131:41361): delete failed for tablet fdf1286333c44421a23476add54d7e01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdf1286333c44421a23476add54d7e01 already in progress: copying tablet
I20260810 18:50:31.216514 1247 raft_consensus.cc:2229] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:31.217338 1247 pending_rounds.cc:62] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: Trying to abort 1 pending ops.
I20260810 18:50:31.217555 1247 pending_rounds.cc:68] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317044556160323584 op_type: NO_OP noop_request { }
I20260810 18:50:31.217741 1250 tablet_replica.cc:330] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05: stopping tablet replica
I20260810 18:50:31.217898 1247 raft_consensus.cc:2866] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260810 18:50:31.218326 1247 raft_consensus.cc:2258] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:31.218611 1250 raft_consensus.cc:2229] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:31.219260 1250 raft_consensus.cc:2258] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:31.221356 1247 ts_tablet_manager.cc:1905] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:50:31.222054 1250 ts_tablet_manager.cc:1905] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:50:31.223155 1245 tablet_bootstrap.cc:492] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260810 18:50:31.224022 1245 tablet_bootstrap.cc:492] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Bootstrap complete.
I20260810 18:50:31.224653 1245 ts_tablet_manager.cc:1397] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260810 18:50:31.227186 1245 raft_consensus.cc:348] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.228060 1245 raft_consensus.cc:729] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9fa5758ce94c5fbd95725a197bcb74, State: Initialized, Role: FOLLOWER
I20260810 18:50:31.228766 1245 consensus_queue.cc:260] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28897e9be92e4a4a8ce102f6421f89bb" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 34549 } } peers { permanent_uuid: "ec9fa5758ce94c5fbd95725a197bcb74" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 41361 } } peers { permanent_uuid: "4b5e867ac5d04ec3a8e336307db60b05" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41211 } }
I20260810 18:50:31.230779 1245 ts_tablet_manager.cc:1428] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
W20260810 18:50:31.231040 1245 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260810 18:50:31.232789 1247 ts_tablet_manager.cc:1918] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 18:50:31.233108 1247 log.cc:1196] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: Deleting WAL directory at /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-1-root/wals/fdf1286333c44421a23476add54d7e01
I20260810 18:50:31.233719 1247 ts_tablet_manager.cc:1939] T fdf1286333c44421a23476add54d7e01 P 28897e9be92e4a4a8ce102f6421f89bb: Deleting consensus metadata
I20260810 18:50:31.234905 727 catalog_manager.cc:4841] TS 28897e9be92e4a4a8ce102f6421f89bb (127.28.93.130:34549): tablet fdf1286333c44421a23476add54d7e01 (table test-table [id=992d2c2f851e4d009812afa279c4d445]) successfully deleted
I20260810 18:50:31.235898 1250 ts_tablet_manager.cc:1918] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260810 18:50:31.236276 1250 log.cc:1196] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05: Deleting WAL directory at /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-0-root/wals/fdf1286333c44421a23476add54d7e01
I20260810 18:50:31.237206 1250 ts_tablet_manager.cc:1939] T fdf1286333c44421a23476add54d7e01 P 4b5e867ac5d04ec3a8e336307db60b05: Deleting consensus metadata
I20260810 18:50:31.236241 1137 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf1286333c44421a23476add54d7e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:50:31 UTC) from {username='slave'} at 127.0.0.1:49956
I20260810 18:50:31.239305 727 catalog_manager.cc:4841] TS 4b5e867ac5d04ec3a8e336307db60b05 (127.28.93.129:41211): tablet fdf1286333c44421a23476add54d7e01 (table test-table [id=992d2c2f851e4d009812afa279c4d445]) successfully deleted
W20260810 18:50:31.240110 727 catalog_manager.cc:4821] TS ec9fa5758ce94c5fbd95725a197bcb74 (127.28.93.131:41361): delete failed for tablet fdf1286333c44421a23476add54d7e01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdf1286333c44421a23476add54d7e01 already in progress: copying tablet
I20260810 18:50:31.282918 1060 tablet_copy_service.cc:342] P 28897e9be92e4a4a8ce102f6421f89bb: Request end of tablet copy session ec9fa5758ce94c5fbd95725a197bcb74-fdf1286333c44421a23476add54d7e01 received from {username='slave'} at 127.0.0.1:42656
I20260810 18:50:31.283391 1060 tablet_copy_service.cc:433] P 28897e9be92e4a4a8ce102f6421f89bb: ending tablet copy session ec9fa5758ce94c5fbd95725a197bcb74-fdf1286333c44421a23476add54d7e01 on tablet fdf1286333c44421a23476add54d7e01 with peer ec9fa5758ce94c5fbd95725a197bcb74
I20260810 18:50:31.286674 1137 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf1286333c44421a23476add54d7e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:50:31 UTC) from {username='slave'} at 127.0.0.1:49956
I20260810 18:50:31.287384 1244 tablet_replica.cc:330] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: stopping tablet replica
I20260810 18:50:31.288021 1244 raft_consensus.cc:2229] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:31.288383 1244 pending_rounds.cc:62] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Trying to abort 1 pending ops.
I20260810 18:50:31.288631 1244 pending_rounds.cc:68] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317044556160323584 op_type: NO_OP noop_request { }
I20260810 18:50:31.288949 1244 raft_consensus.cc:2866] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260810 18:50:31.289232 1244 raft_consensus.cc:2258] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:31.291251 1244 ts_tablet_manager.cc:1905] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:50:31.300427 1244 ts_tablet_manager.cc:1918] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260810 18:50:31.300714 1244 log.cc:1196] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Deleting WAL directory at /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786387803638833-29046-0/minicluster-data/ts-2-root/wals/fdf1286333c44421a23476add54d7e01
I20260810 18:50:31.301251 1244 ts_tablet_manager.cc:1939] T fdf1286333c44421a23476add54d7e01 P ec9fa5758ce94c5fbd95725a197bcb74: Deleting consensus metadata
I20260810 18:50:31.303323 727 catalog_manager.cc:4841] TS ec9fa5758ce94c5fbd95725a197bcb74 (127.28.93.131:41361): tablet fdf1286333c44421a23476add54d7e01 (table test-table [id=992d2c2f851e4d009812afa279c4d445]) successfully deleted
I20260810 18:50:31.347330 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:31.365952 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:31.381757 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:31.389789 29046 tablet_server.cc:178] TabletServer@127.28.93.130:0 shutting down...
I20260810 18:50:31.405491 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:31.422073 29046 tablet_server.cc:194] TabletServer@127.28.93.130:0 shutdown complete.
I20260810 18:50:31.430229 29046 tablet_server.cc:178] TabletServer@127.28.93.131:0 shutting down...
I20260810 18:50:31.447247 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:31.466252 29046 tablet_server.cc:194] TabletServer@127.28.93.131:0 shutdown complete.
I20260810 18:50:31.475023 29046 master.cc:537] Master@127.28.93.190:40151 shutting down...
I20260810 18:50:31.488967 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:31.489537 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:31.489842 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2bef2addf1a54743bec9bcd3c02215c7: stopping tablet replica
I20260810 18:50:31.507332 29046 master.cc:552] Master@127.28.93.190:40151 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1084 ms)
[----------] 4 tests from TsTabletManagerITest (16088 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20260810 18:50:31.532924 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:44667
I20260810 18:50:31.533941 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:31.538527 1260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:31.539885 1261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:31.541666 1268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:31.541808 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:31.542696 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:31.542862 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:31.543016 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387831543003 us; error 0 us; skew 500 ppm
I20260810 18:50:31.543521 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:31.545640 29046 webserver.cc:460] Webserver started at http://127.28.93.190:43977/ using document root <none> and password file <none>
I20260810 18:50:31.546082 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:31.546252 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:31.546509 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:31.547520 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "87c61c8af35b4879aa44ccc833a83763"
format_stamp: "Formatted at 2026-08-10 18:50:31 on dist-test-slave-nj21"
I20260810 18:50:31.551614 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:50:31.554682 1273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:31.555330 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:31.555594 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "87c61c8af35b4879aa44ccc833a83763"
format_stamp: "Formatted at 2026-08-10 18:50:31 on dist-test-slave-nj21"
I20260810 18:50:31.555886 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:31.566129 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:31.567014 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:31.603415 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:44667
I20260810 18:50:31.603504 1362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:44667 every 8 connection(s)
I20260810 18:50:31.607005 1363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:31.616149 1363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763: Bootstrap starting.
I20260810 18:50:31.620174 1363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:31.623807 1363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763: No bootstrap required, opened a new log
I20260810 18:50:31.625568 1363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c61c8af35b4879aa44ccc833a83763" member_type: VOTER }
I20260810 18:50:31.625955 1363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:31.626161 1363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c61c8af35b4879aa44ccc833a83763, State: Initialized, Role: FOLLOWER
I20260810 18:50:31.626715 1363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c61c8af35b4879aa44ccc833a83763" member_type: VOTER }
I20260810 18:50:31.627177 1363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:31.627388 1363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:31.627635 1363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:31.631924 1363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c61c8af35b4879aa44ccc833a83763" member_type: VOTER }
I20260810 18:50:31.632383 1363 leader_election.cc:302] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c61c8af35b4879aa44ccc833a83763; no voters:
I20260810 18:50:31.633437 1363 leader_election.cc:288] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:31.633733 1367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:31.634932 1367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 1 LEADER]: Becoming Leader. State: Replica: 87c61c8af35b4879aa44ccc833a83763, State: Running, Role: LEADER
I20260810 18:50:31.635527 1367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c61c8af35b4879aa44ccc833a83763" member_type: VOTER }
I20260810 18:50:31.636134 1363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:31.638381 1368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87c61c8af35b4879aa44ccc833a83763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c61c8af35b4879aa44ccc833a83763" member_type: VOTER } }
I20260810 18:50:31.639200 1368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:31.638459 1369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c61c8af35b4879aa44ccc833a83763. Latest consensus state: current_term: 1 leader_uuid: "87c61c8af35b4879aa44ccc833a83763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c61c8af35b4879aa44ccc833a83763" member_type: VOTER } }
I20260810 18:50:31.640491 1369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:31.641868 1378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:31.646239 1378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:31.649740 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:31.654287 1378 catalog_manager.cc:1347] Generated new cluster ID: 773d0b7e23cb4493b8fbe198852c9d7c
I20260810 18:50:31.654562 1378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:31.667415 1378 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:31.668947 1378 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:31.688177 1378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763: Generated new TSK 0
I20260810 18:50:31.688794 1378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:31.715771 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:31.721256 1393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:31.722685 1394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:31.724895 1397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:31.725572 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:31.726325 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:31.726496 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:31.726639 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387831726625 us; error 0 us; skew 500 ppm
I20260810 18:50:31.727118 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:31.731799 29046 webserver.cc:460] Webserver started at http://127.28.93.129:43279/ using document root <none> and password file <none>
I20260810 18:50:31.732259 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:31.732432 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:31.732669 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:31.733682 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "929ef633a1bf46c2bae9d5e26bf23505"
format_stamp: "Formatted at 2026-08-10 18:50:31 on dist-test-slave-nj21"
I20260810 18:50:31.737991 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:50:31.740957 1404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:31.741761 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:50:31.742033 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "929ef633a1bf46c2bae9d5e26bf23505"
format_stamp: "Formatted at 2026-08-10 18:50:31 on dist-test-slave-nj21"
I20260810 18:50:31.742292 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:31.766117 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:31.767059 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:31.772058 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:31.772354 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:31.772662 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:31.772856 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:31.822206 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:39705
I20260810 18:50:31.822309 1519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:39705 every 8 connection(s)
I20260810 18:50:31.826592 1525 heartbeater.cc:347] Connected to a master server at 127.28.93.190:44667
I20260810 18:50:31.826902 1525 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:31.827538 1525 heartbeater.cc:511] Master 127.28.93.190:44667 requested a full tablet report, sending...
I20260810 18:50:31.829290 1301 ts_manager.cc:194] Registered new tserver with Master: 929ef633a1bf46c2bae9d5e26bf23505 (127.28.93.129:39705)
I20260810 18:50:31.829995 29046 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004837521s
I20260810 18:50:31.831802 1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51434
I20260810 18:50:31.864694 1462 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260810 18:50:31.865969 1462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:31.877291 1543 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: Bootstrap starting.
I20260810 18:50:31.881544 1543 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:31.885703 1543 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: No bootstrap required, opened a new log
I20260810 18:50:31.886075 1543 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260810 18:50:31.888088 1543 raft_consensus.cc:348] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } }
I20260810 18:50:31.888495 1543 raft_consensus.cc:374] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:31.888743 1543 raft_consensus.cc:729] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929ef633a1bf46c2bae9d5e26bf23505, State: Initialized, Role: FOLLOWER
I20260810 18:50:31.889218 1543 consensus_queue.cc:260] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } }
I20260810 18:50:31.889735 1543 raft_consensus.cc:388] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:31.889930 1543 raft_consensus.cc:482] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:31.890141 1543 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:31.894366 1543 raft_consensus.cc:504] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } }
I20260810 18:50:31.894893 1543 leader_election.cc:302] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 929ef633a1bf46c2bae9d5e26bf23505; no voters:
I20260810 18:50:31.896046 1543 leader_election.cc:288] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:31.896399 1549 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:31.897717 1549 raft_consensus.cc:686] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 1 LEADER]: Becoming Leader. State: Replica: 929ef633a1bf46c2bae9d5e26bf23505, State: Running, Role: LEADER
I20260810 18:50:31.898041 1525 heartbeater.cc:503] Master 127.28.93.190:44667 was elected leader, sending a full tablet report...
I20260810 18:50:31.898509 1543 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260810 18:50:31.898418 1549 consensus_queue.cc:237] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } }
I20260810 18:50:31.902611 1551 tablet_replica.cc:439] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "929ef633a1bf46c2bae9d5e26bf23505" committed_config { opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } } }
I20260810 18:50:31.903407 1551 tablet_replica.cc:442] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:50:31.903802 1554 tablet_replica.cc:439] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: TxnStatusTablet state changed. Reason: New leader 929ef633a1bf46c2bae9d5e26bf23505. Latest consensus state: current_term: 1 leader_uuid: "929ef633a1bf46c2bae9d5e26bf23505" committed_config { opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } } }
I20260810 18:50:31.904291 1554 tablet_replica.cc:442] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: This TxnStatusTablet replica's current role is: LEADER
W20260810 18:50:31.904997 1301 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:31.905391 1559 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:50:31.905892 1559 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260810 18:50:31.908891 1301 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:31.941498 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:31.963905 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:31.965041 29046 tablet_replica.cc:330] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: stopping tablet replica
I20260810 18:50:31.965883 29046 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:31.966449 29046 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:31.988771 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:31.999418 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:32.004920 1564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:32.005685 1565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:32.008298 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:32.008484 1569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:32.009464 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:32.009639 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:32.009784 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387832009773 us; error 0 us; skew 500 ppm
I20260810 18:50:32.010283 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:32.012984 29046 webserver.cc:460] Webserver started at http://127.28.93.129:43279/ using document root <none> and password file <none>
I20260810 18:50:32.013448 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:32.013600 29046 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260810 18:50:32.017278 29046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:50:32.020037 1574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:32.020831 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:32.021166 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "929ef633a1bf46c2bae9d5e26bf23505"
format_stamp: "Formatted at 2026-08-10 18:50:31 on dist-test-slave-nj21"
I20260810 18:50:32.021531 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:32.035142 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:32.036399 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:32.044328 1587 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 18:50:32.051280 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:50:32.051549 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20260810 18:50:32.051899 29046 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:50:32.056728 29046 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:50:32.056951 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20260810 18:50:32.057308 1587 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: Bootstrap starting.
I20260810 18:50:32.105063 1587 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:32.106092 1587 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: Bootstrap complete.
I20260810 18:50:32.106809 1587 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: Time spent bootstrapping tablet: real 0.050s user 0.021s sys 0.023s
I20260810 18:50:32.109360 1587 raft_consensus.cc:348] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } }
I20260810 18:50:32.109975 1587 raft_consensus.cc:729] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929ef633a1bf46c2bae9d5e26bf23505, State: Initialized, Role: FOLLOWER
I20260810 18:50:32.110787 1587 consensus_queue.cc:260] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } }
I20260810 18:50:32.111456 1587 raft_consensus.cc:388] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:32.111770 1587 raft_consensus.cc:482] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:32.112099 1587 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:32.119031 1587 raft_consensus.cc:504] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } }
I20260810 18:50:32.119783 1587 leader_election.cc:302] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 929ef633a1bf46c2bae9d5e26bf23505; no voters:
I20260810 18:50:32.121493 1587 leader_election.cc:288] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:50:32.121877 1607 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:32.127787 1587 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505: Time spent starting tablet: real 0.021s user 0.019s sys 0.004s
I20260810 18:50:32.136128 1607 raft_consensus.cc:686] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 2 LEADER]: Becoming Leader. State: Replica: 929ef633a1bf46c2bae9d5e26bf23505, State: Running, Role: LEADER
I20260810 18:50:32.137030 1607 consensus_queue.cc:237] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } }
I20260810 18:50:32.140203 1610 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "929ef633a1bf46c2bae9d5e26bf23505" committed_config { opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } } }
I20260810 18:50:32.140766 1610 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260810 18:50:32.142454 1611 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 929ef633a1bf46c2bae9d5e26bf23505. Latest consensus state: current_term: 2 leader_uuid: "929ef633a1bf46c2bae9d5e26bf23505" committed_config { opid_index: -1 peers { permanent_uuid: "929ef633a1bf46c2bae9d5e26bf23505" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 39705 } } }
I20260810 18:50:32.144177 1611 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260810 18:50:32.146170 1633 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:50:32.146649 1633 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:50:32.177255 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:39705
I20260810 18:50:32.177321 1699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:39705 every 8 connection(s)
I20260810 18:50:32.183658 1700 heartbeater.cc:347] Connected to a master server at 127.28.93.190:44667
I20260810 18:50:32.184060 1700 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:32.184737 1700 heartbeater.cc:511] Master 127.28.93.190:44667 requested a full tablet report, sending...
I20260810 18:50:32.189424 1301 ts_manager.cc:194] Re-registered known tserver with Master: 929ef633a1bf46c2bae9d5e26bf23505 (127.28.93.129:39705)
I20260810 18:50:32.189743 29046 tablet_server.cc:178] TabletServer@127.28.93.129:39705 shutting down...
W20260810 18:50:32.189915 1301 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:32.192580 1301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51450
I20260810 18:50:32.196182 1700 heartbeater.cc:503] Master 127.28.93.190:44667 was elected leader, sending a full tablet report...
W20260810 18:50:32.199210 1301 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:32.215626 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:32.216446 29046 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:32.217078 29046 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 2 LEADER]: Raft consensus shutting down.
I20260810 18:50:32.217573 29046 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 929ef633a1bf46c2bae9d5e26bf23505 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:32.238612 29046 tablet_server.cc:194] TabletServer@127.28.93.129:39705 shutdown complete.
I20260810 18:50:32.250536 29046 master.cc:537] Master@127.28.93.190:44667 shutting down...
I20260810 18:50:32.265609 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:32.266074 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:32.266352 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P 87c61c8af35b4879aa44ccc833a83763: stopping tablet replica
I20260810 18:50:32.283823 29046 master.cc:552] Master@127.28.93.190:44667 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (773 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20260810 18:50:32.305640 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:46065
I20260810 18:50:32.306617 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:32.311630 1707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:32.313577 1709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:32.314318 1715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:32.315412 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:32.316116 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:32.316329 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:32.316476 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387832316465 us; error 0 us; skew 500 ppm
I20260810 18:50:32.317008 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:32.319489 29046 webserver.cc:460] Webserver started at http://127.28.93.190:38305/ using document root <none> and password file <none>
I20260810 18:50:32.319960 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:32.320122 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:32.320377 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:32.321419 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "584e9bad8bb74996a09bc6f64886bb42"
format_stamp: "Formatted at 2026-08-10 18:50:32 on dist-test-slave-nj21"
I20260810 18:50:32.326018 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:50:32.329370 1721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:32.330170 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:50:32.330433 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "584e9bad8bb74996a09bc6f64886bb42"
format_stamp: "Formatted at 2026-08-10 18:50:32 on dist-test-slave-nj21"
I20260810 18:50:32.330694 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:32.346783 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:32.347723 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:32.391290 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:46065
I20260810 18:50:32.391389 1818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:46065 every 8 connection(s)
I20260810 18:50:32.395470 1819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:32.405071 1819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42: Bootstrap starting.
I20260810 18:50:32.409108 1819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:32.413028 1819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42: No bootstrap required, opened a new log
I20260810 18:50:32.414806 1819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584e9bad8bb74996a09bc6f64886bb42" member_type: VOTER }
I20260810 18:50:32.415176 1819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:32.415388 1819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584e9bad8bb74996a09bc6f64886bb42, State: Initialized, Role: FOLLOWER
I20260810 18:50:32.415943 1819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584e9bad8bb74996a09bc6f64886bb42" member_type: VOTER }
I20260810 18:50:32.416370 1819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:32.416584 1819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:32.416826 1819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:32.420881 1819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584e9bad8bb74996a09bc6f64886bb42" member_type: VOTER }
I20260810 18:50:32.421344 1819 leader_election.cc:302] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 584e9bad8bb74996a09bc6f64886bb42; no voters:
I20260810 18:50:32.422577 1819 leader_election.cc:288] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:32.422881 1824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:32.424544 1824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 1 LEADER]: Becoming Leader. State: Replica: 584e9bad8bb74996a09bc6f64886bb42, State: Running, Role: LEADER
I20260810 18:50:32.425318 1824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584e9bad8bb74996a09bc6f64886bb42" member_type: VOTER }
I20260810 18:50:32.425819 1819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:32.428215 1825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "584e9bad8bb74996a09bc6f64886bb42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584e9bad8bb74996a09bc6f64886bb42" member_type: VOTER } }
I20260810 18:50:32.428292 1826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 584e9bad8bb74996a09bc6f64886bb42. Latest consensus state: current_term: 1 leader_uuid: "584e9bad8bb74996a09bc6f64886bb42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "584e9bad8bb74996a09bc6f64886bb42" member_type: VOTER } }
I20260810 18:50:32.428833 1825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:32.428903 1826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:32.442692 1831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:32.449532 1831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:32.451853 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:32.459180 1831 catalog_manager.cc:1347] Generated new cluster ID: b7dfb6f2121d4dce8cc3d75eacfb3353
I20260810 18:50:32.459409 1831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:32.479662 1831 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:32.481297 1831 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:32.491915 1831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42: Generated new TSK 0
I20260810 18:50:32.492517 1831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:32.518356 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:32.525151 1848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:32.526285 1849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:32.529189 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:32.529258 1851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:32.530261 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:32.530462 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:32.530632 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387832530617 us; error 0 us; skew 500 ppm
I20260810 18:50:32.531163 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:32.533879 29046 webserver.cc:460] Webserver started at http://127.28.93.129:34909/ using document root <none> and password file <none>
I20260810 18:50:32.534346 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:32.534523 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:32.534767 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:32.535876 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "8d81762ee685466d956a60e8500a1f40"
format_stamp: "Formatted at 2026-08-10 18:50:32 on dist-test-slave-nj21"
I20260810 18:50:32.540428 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:50:32.543715 1857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:32.544520 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:32.544814 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "8d81762ee685466d956a60e8500a1f40"
format_stamp: "Formatted at 2026-08-10 18:50:32 on dist-test-slave-nj21"
I20260810 18:50:32.545087 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:32.584640 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:32.585644 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:32.605639 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:32.605984 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:32.606287 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:32.606534 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:32.676940 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:41021
I20260810 18:50:32.677074 1972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:41021 every 8 connection(s)
I20260810 18:50:32.682034 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:50:32.682075 1973 heartbeater.cc:347] Connected to a master server at 127.28.93.190:46065
I20260810 18:50:32.682610 1973 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:32.683391 1973 heartbeater.cc:511] Master 127.28.93.190:46065 requested a full tablet report, sending...
I20260810 18:50:32.685570 1748 ts_manager.cc:194] Registered new tserver with Master: 8d81762ee685466d956a60e8500a1f40 (127.28.93.129:41021)
W20260810 18:50:32.688501 1976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:32.688781 1748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57054
W20260810 18:50:32.692082 1977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:32.695384 1986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:32.695433 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:32.696736 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:32.696995 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:32.697187 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387832697174 us; error 0 us; skew 500 ppm
I20260810 18:50:32.697850 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:32.700526 29046 webserver.cc:460] Webserver started at http://127.28.93.130:35075/ using document root <none> and password file <none>
I20260810 18:50:32.701112 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:32.701356 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:32.701659 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:32.703089 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-1-root/instance:
uuid: "37186da7a267467393fb96c57ad80273"
format_stamp: "Formatted at 2026-08-10 18:50:32 on dist-test-slave-nj21"
I20260810 18:50:32.708796 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 18:50:32.712839 1994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:32.713652 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260810 18:50:32.713976 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-1-root
uuid: "37186da7a267467393fb96c57ad80273"
format_stamp: "Formatted at 2026-08-10 18:50:32 on dist-test-slave-nj21"
I20260810 18:50:32.714320 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:32.732574 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:32.733584 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:32.738179 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:32.738445 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:32.738729 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:32.738945 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:32.810840 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.130:43809
I20260810 18:50:32.810949 2112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.130:43809 every 8 connection(s)
I20260810 18:50:32.815583 2113 heartbeater.cc:347] Connected to a master server at 127.28.93.190:46065
I20260810 18:50:32.816028 2113 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:32.816886 2113 heartbeater.cc:511] Master 127.28.93.190:46065 requested a full tablet report, sending...
I20260810 18:50:32.818866 1748 ts_manager.cc:194] Registered new tserver with Master: 37186da7a267467393fb96c57ad80273 (127.28.93.130:43809)
I20260810 18:50:32.819159 29046 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.005008678s
I20260810 18:50:32.821022 1748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57078
I20260810 18:50:32.857728 1908 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260810 18:50:32.858903 1908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:32.869419 2132 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: Bootstrap starting.
I20260810 18:50:32.873654 2132 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:32.877730 2132 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: No bootstrap required, opened a new log
I20260810 18:50:32.878091 2132 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260810 18:50:32.879886 2132 raft_consensus.cc:348] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } }
I20260810 18:50:32.880296 2132 raft_consensus.cc:374] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:32.880523 2132 raft_consensus.cc:729] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d81762ee685466d956a60e8500a1f40, State: Initialized, Role: FOLLOWER
I20260810 18:50:32.881008 2132 consensus_queue.cc:260] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } }
I20260810 18:50:32.881469 2132 raft_consensus.cc:388] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:32.881681 2132 raft_consensus.cc:482] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:32.881954 2132 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:32.886150 2132 raft_consensus.cc:504] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } }
I20260810 18:50:32.886646 2132 leader_election.cc:302] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d81762ee685466d956a60e8500a1f40; no voters:
I20260810 18:50:32.887948 2132 leader_election.cc:288] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:32.888331 2137 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:32.889870 1973 heartbeater.cc:503] Master 127.28.93.190:46065 was elected leader, sending a full tablet report...
I20260810 18:50:32.890314 2132 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260810 18:50:32.891103 2137 raft_consensus.cc:686] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 1 LEADER]: Becoming Leader. State: Replica: 8d81762ee685466d956a60e8500a1f40, State: Running, Role: LEADER
I20260810 18:50:32.891696 2137 consensus_queue.cc:237] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } }
I20260810 18:50:32.895789 2139 tablet_replica.cc:439] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: TxnStatusTablet state changed. Reason: New leader 8d81762ee685466d956a60e8500a1f40. Latest consensus state: current_term: 1 leader_uuid: "8d81762ee685466d956a60e8500a1f40" committed_config { opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } } }
I20260810 18:50:32.896423 2139 tablet_replica.cc:442] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: This TxnStatusTablet replica's current role is: LEADER
W20260810 18:50:32.897989 1748 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:32.900005 2138 tablet_replica.cc:439] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d81762ee685466d956a60e8500a1f40" committed_config { opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } } }
I20260810 18:50:32.900554 2138 tablet_replica.cc:442] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:50:32.904026 2144 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:50:32.904438 2144 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260810 18:50:32.908102 1748 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:32.960247 2153 ts_tablet_manager.cc:927] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: Initiating tablet copy from peer 8d81762ee685466d956a60e8500a1f40 (127.28.93.129:41021)
I20260810 18:50:32.961946 2153 tablet_copy_client.cc:316] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: tablet copy: Beginning tablet copy session from remote peer at address 127.28.93.129:41021
I20260810 18:50:32.971643 1944 tablet_copy_service.cc:140] P 8d81762ee685466d956a60e8500a1f40: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 37186da7a267467393fb96c57ad80273 ({username='slave'} at 127.0.0.1:50310)
I20260810 18:50:32.972198 1944 tablet_copy_service.cc:161] P 8d81762ee685466d956a60e8500a1f40: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 37186da7a267467393fb96c57ad80273 at {username='slave'} at 127.0.0.1:50310: session id = 37186da7a267467393fb96c57ad80273-11111111111111111111111111111111
I20260810 18:50:32.979257 1944 tablet_copy_source_session.cc:205] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:50:32.981912 2153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:32.992404 2153 tablet_copy_client.cc:791] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: tablet copy: Starting download of 0 data blocks...
I20260810 18:50:32.992985 2153 tablet_copy_client.cc:655] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: tablet copy: Starting download of 1 WAL segments...
I20260810 18:50:32.996305 2153 tablet_copy_client.cc:531] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:50:33.001530 2153 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: Bootstrap starting.
I20260810 18:50:33.021313 2153 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:33.022058 2153 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: Bootstrap complete.
I20260810 18:50:33.022670 2153 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20260810 18:50:33.024981 2153 raft_consensus.cc:348] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } }
I20260810 18:50:33.025542 2153 raft_consensus.cc:729] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 37186da7a267467393fb96c57ad80273, State: Initialized, Role: NON_PARTICIPANT
I20260810 18:50:33.026187 2153 consensus_queue.cc:260] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } }
I20260810 18:50:33.028311 2156 tablet_replica.cc:439] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } } }
I20260810 18:50:33.028584 2113 heartbeater.cc:503] Master 127.28.93.190:46065 was elected leader, sending a full tablet report...
I20260810 18:50:33.029021 2156 tablet_replica.cc:442] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20260810 18:50:33.029217 2153 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: Time spent starting tablet: real 0.006s user 0.007s sys 0.002s
I20260810 18:50:33.031417 1944 tablet_copy_service.cc:342] P 8d81762ee685466d956a60e8500a1f40: Request end of tablet copy session 37186da7a267467393fb96c57ad80273-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:50310
I20260810 18:50:33.031877 1944 tablet_copy_service.cc:433] P 8d81762ee685466d956a60e8500a1f40: ending tablet copy session 37186da7a267467393fb96c57ad80273-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 37186da7a267467393fb96c57ad80273
W20260810 18:50:33.032155 1748 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260810 18:50:33.036835 1748 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:33.057394 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:33.081358 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:33.082340 29046 tablet_replica.cc:330] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: stopping tablet replica
I20260810 18:50:33.083001 29046 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:33.083539 29046 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 18:50:34.334689 29046 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 1969)
W20260810 18:50:34.338449 1969 debug-util.cc:398] Leaking SignalData structure 0x7b080006d8c0 after lost signal to thread 29051
W20260810 18:50:34.339496 1969 debug-util.cc:398] Leaking SignalData structure 0x7b080018c460 after lost signal to thread 1818
W20260810 18:50:34.340896 1969 debug-util.cc:398] Leaking SignalData structure 0x7b0800104120 after lost signal to thread 2112
W20260810 18:50:34.341212 1969 debug-util.cc:398] Leaking SignalData structure 0x7b080003ef60 after lost signal to thread 2143
W20260810 18:50:35.335245 29046 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 1969)
W20260810 18:50:36.335752 29046 thread.cc:536] Waited for 3000ms trying to join with diag-logger (tid 1969)
I20260810 18:50:36.901592 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:36.917768 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:36.924440 2159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:36.924607 2160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:36.927076 2162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:36.927176 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:36.928377 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:36.928565 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:36.928720 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387836928710 us; error 0 us; skew 500 ppm
I20260810 18:50:36.929180 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:36.935987 29046 webserver.cc:460] Webserver started at http://127.28.93.129:34909/ using document root <none> and password file <none>
I20260810 18:50:36.936436 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:36.936581 29046 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260810 18:50:36.940158 29046 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 18:50:36.942772 2171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:36.943452 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 18:50:36.943753 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "8d81762ee685466d956a60e8500a1f40"
format_stamp: "Formatted at 2026-08-10 18:50:32 on dist-test-slave-nj21"
I20260810 18:50:36.944160 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:36.979450 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:36.980692 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:36.988090 2190 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 18:50:36.994630 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 18:50:36.994875 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.001s
I20260810 18:50:36.995143 29046 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 18:50:37.000075 29046 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 18:50:37.000403 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20260810 18:50:37.000500 2190 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: Bootstrap starting.
I20260810 18:50:37.031731 2190 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:37.032788 2190 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: Bootstrap complete.
I20260810 18:50:37.033519 2190 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.005s
I20260810 18:50:37.036743 2190 raft_consensus.cc:348] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } }
I20260810 18:50:37.037896 2190 raft_consensus.cc:729] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d81762ee685466d956a60e8500a1f40, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.038691 2190 consensus_queue.cc:260] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } }
I20260810 18:50:37.039417 2190 raft_consensus.cc:388] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:37.039799 2190 raft_consensus.cc:482] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:37.040180 2190 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:37.048163 2190 raft_consensus.cc:504] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } }
I20260810 18:50:37.048866 2190 leader_election.cc:302] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d81762ee685466d956a60e8500a1f40; no voters:
I20260810 18:50:37.050211 2190 leader_election.cc:288] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:50:37.050495 2269 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:37.053061 2190 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
I20260810 18:50:37.054730 2269 raft_consensus.cc:686] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 2 LEADER]: Becoming Leader. State: Replica: 8d81762ee685466d956a60e8500a1f40, State: Running, Role: LEADER
I20260810 18:50:37.055717 2269 consensus_queue.cc:237] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } }
I20260810 18:50:37.066210 2273 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8d81762ee685466d956a60e8500a1f40" committed_config { opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } } }
I20260810 18:50:37.066881 2273 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260810 18:50:37.066851 2274 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 8d81762ee685466d956a60e8500a1f40. Latest consensus state: current_term: 2 leader_uuid: "8d81762ee685466d956a60e8500a1f40" committed_config { opid_index: -1 peers { permanent_uuid: "8d81762ee685466d956a60e8500a1f40" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 41021 } } }
I20260810 18:50:37.067435 2274 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260810 18:50:37.068661 2294 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:50:37.069129 2294 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:50:37.069895 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:41021
I20260810 18:50:37.070163 2296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:41021 every 8 connection(s)
I20260810 18:50:37.075515 2298 heartbeater.cc:347] Connected to a master server at 127.28.93.190:46065
I20260810 18:50:37.075949 2298 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:37.076735 2298 heartbeater.cc:511] Master 127.28.93.190:46065 requested a full tablet report, sending...
I20260810 18:50:37.079362 1748 ts_manager.cc:194] Re-registered known tserver with Master: 8d81762ee685466d956a60e8500a1f40 (127.28.93.129:41021)
W20260810 18:50:37.080155 1748 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:37.080775 29046 tablet_server.cc:178] TabletServer@127.28.93.129:41021 shutting down...
I20260810 18:50:37.085083 1748 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45156
I20260810 18:50:37.089596 2298 heartbeater.cc:503] Master 127.28.93.190:46065 was elected leader, sending a full tablet report...
W20260810 18:50:37.092447 1748 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:37.105536 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:37.106319 29046 tablet_replica.cc:330] stopping tablet replica
I20260810 18:50:37.106994 29046 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 2 LEADER]: Raft consensus shutting down.
I20260810 18:50:37.107391 29046 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 8d81762ee685466d956a60e8500a1f40 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:37.126621 29046 tablet_server.cc:194] TabletServer@127.28.93.129:41021 shutdown complete.
I20260810 18:50:37.136689 29046 tablet_server.cc:178] TabletServer@127.28.93.130:0 shutting down...
I20260810 18:50:37.154238 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:37.154896 29046 tablet_replica.cc:330] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273: stopping tablet replica
I20260810 18:50:37.155467 29046 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260810 18:50:37.155929 29046 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 37186da7a267467393fb96c57ad80273 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260810 18:50:37.173836 29046 tablet_server.cc:194] TabletServer@127.28.93.130:0 shutdown complete.
I20260810 18:50:37.184844 29046 master.cc:537] Master@127.28.93.190:46065 shutting down...
I20260810 18:50:37.202052 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:37.202651 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:37.202980 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P 584e9bad8bb74996a09bc6f64886bb42: stopping tablet replica
I20260810 18:50:37.220913 29046 master.cc:552] Master@127.28.93.190:46065 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (4943 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20260810 18:50:37.249190 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:41787
I20260810 18:50:37.250331 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:37.255331 2309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:37.256462 2310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:37.257714 2314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:37.258162 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:37.258947 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:37.259114 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:37.259265 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387837259251 us; error 0 us; skew 500 ppm
I20260810 18:50:37.259824 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:37.262030 29046 webserver.cc:460] Webserver started at http://127.28.93.190:36329/ using document root <none> and password file <none>
I20260810 18:50:37.262495 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:37.262657 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:37.262913 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:37.264074 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "439bc00bf2c443d984d2d56ddfe89aa7"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-nj21"
I20260810 18:50:37.268337 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 18:50:37.271406 2320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.272122 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:37.272377 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "439bc00bf2c443d984d2d56ddfe89aa7"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-nj21"
I20260810 18:50:37.272643 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:37.287305 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:37.288390 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:37.325067 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:41787
I20260810 18:50:37.325176 2417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:41787 every 8 connection(s)
I20260810 18:50:37.328711 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.338480 2418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7: Bootstrap starting.
I20260810 18:50:37.342700 2418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.346668 2418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7: No bootstrap required, opened a new log
I20260810 18:50:37.348611 2418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439bc00bf2c443d984d2d56ddfe89aa7" member_type: VOTER }
I20260810 18:50:37.348978 2418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.349181 2418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439bc00bf2c443d984d2d56ddfe89aa7, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.349751 2418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439bc00bf2c443d984d2d56ddfe89aa7" member_type: VOTER }
I20260810 18:50:37.350206 2418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:37.350414 2418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:37.350637 2418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:37.354661 2418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439bc00bf2c443d984d2d56ddfe89aa7" member_type: VOTER }
I20260810 18:50:37.355139 2418 leader_election.cc:302] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 439bc00bf2c443d984d2d56ddfe89aa7; no voters:
I20260810 18:50:37.356302 2418 leader_election.cc:288] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:37.356663 2422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:37.357877 2422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 1 LEADER]: Becoming Leader. State: Replica: 439bc00bf2c443d984d2d56ddfe89aa7, State: Running, Role: LEADER
I20260810 18:50:37.358502 2422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439bc00bf2c443d984d2d56ddfe89aa7" member_type: VOTER }
I20260810 18:50:37.358956 2418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:37.362305 2425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439bc00bf2c443d984d2d56ddfe89aa7. Latest consensus state: current_term: 1 leader_uuid: "439bc00bf2c443d984d2d56ddfe89aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439bc00bf2c443d984d2d56ddfe89aa7" member_type: VOTER } }
I20260810 18:50:37.362474 2423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "439bc00bf2c443d984d2d56ddfe89aa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439bc00bf2c443d984d2d56ddfe89aa7" member_type: VOTER } }
I20260810 18:50:37.362890 2425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:37.363073 2423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:37.365324 2435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:37.369906 2435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:37.373513 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:37.378264 2435 catalog_manager.cc:1347] Generated new cluster ID: 00d60426c87a46229d4b2517c1559d27
I20260810 18:50:37.378466 2435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:37.396849 2435 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:37.398413 2435 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:37.409576 2435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7: Generated new TSK 0
I20260810 18:50:37.410167 2435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:37.439730 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:37.445371 2452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:37.446406 2454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:37.447937 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:37.448058 2458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:37.448920 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:37.449106 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:37.449258 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387837449244 us; error 0 us; skew 500 ppm
I20260810 18:50:37.449787 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:37.452028 29046 webserver.cc:460] Webserver started at http://127.28.93.129:35839/ using document root <none> and password file <none>
I20260810 18:50:37.452481 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:37.452653 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:37.452898 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:37.453910 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "f4f276754af04e6e9c745ce1ad4eaa94"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-nj21"
I20260810 18:50:37.458055 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 18:50:37.461321 2464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.462078 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:50:37.462373 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "f4f276754af04e6e9c745ce1ad4eaa94"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-nj21"
I20260810 18:50:37.462656 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:37.480530 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:37.481596 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:37.486289 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:37.486593 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 18:50:37.486915 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:37.487139 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.542227 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:43461
I20260810 18:50:37.542322 2579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:43461 every 8 connection(s)
I20260810 18:50:37.546859 2580 heartbeater.cc:347] Connected to a master server at 127.28.93.190:41787
I20260810 18:50:37.547204 2580 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:37.547940 2580 heartbeater.cc:511] Master 127.28.93.190:41787 requested a full tablet report, sending...
I20260810 18:50:37.549768 2351 ts_manager.cc:194] Registered new tserver with Master: f4f276754af04e6e9c745ce1ad4eaa94 (127.28.93.129:43461)
I20260810 18:50:37.550215 29046 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004878566s
I20260810 18:50:37.552824 2351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42214
I20260810 18:50:37.585273 2527 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260810 18:50:37.586409 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.597256 2599 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94: Bootstrap starting.
I20260810 18:50:37.601867 2599 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.605574 2599 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94: No bootstrap required, opened a new log
I20260810 18:50:37.605927 2599 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260810 18:50:37.607712 2599 raft_consensus.cc:348] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "f4f276754af04e6e9c745ce1ad4eaa94" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43461 } }
I20260810 18:50:37.608109 2599 raft_consensus.cc:374] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.608314 2599 raft_consensus.cc:729] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f276754af04e6e9c745ce1ad4eaa94, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.608781 2599 consensus_queue.cc:260] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "f4f276754af04e6e9c745ce1ad4eaa94" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43461 } }
I20260810 18:50:37.609223 2599 raft_consensus.cc:388] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:37.609457 2599 raft_consensus.cc:482] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:37.609689 2599 raft_consensus.cc:3037] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:37.614449 2599 raft_consensus.cc:504] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "f4f276754af04e6e9c745ce1ad4eaa94" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43461 } }
I20260810 18:50:37.614941 2599 leader_election.cc:302] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4f276754af04e6e9c745ce1ad4eaa94; no voters:
I20260810 18:50:37.616022 2599 leader_election.cc:288] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:37.616463 2602 raft_consensus.cc:2781] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:37.617977 2580 heartbeater.cc:503] Master 127.28.93.190:41787 was elected leader, sending a full tablet report...
I20260810 18:50:37.618173 2599 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260810 18:50:37.618593 2602 raft_consensus.cc:686] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 1 LEADER]: Becoming Leader. State: Replica: f4f276754af04e6e9c745ce1ad4eaa94, State: Running, Role: LEADER
I20260810 18:50:37.619227 2602 consensus_queue.cc:237] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "f4f276754af04e6e9c745ce1ad4eaa94" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43461 } }
I20260810 18:50:37.623199 2603 tablet_replica.cc:439] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4f276754af04e6e9c745ce1ad4eaa94" committed_config { opid_index: -1 peers { permanent_uuid: "f4f276754af04e6e9c745ce1ad4eaa94" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43461 } } }
I20260810 18:50:37.623765 2603 tablet_replica.cc:442] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:50:37.624027 2604 tablet_replica.cc:439] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94: TxnStatusTablet state changed. Reason: New leader f4f276754af04e6e9c745ce1ad4eaa94. Latest consensus state: current_term: 1 leader_uuid: "f4f276754af04e6e9c745ce1ad4eaa94" committed_config { opid_index: -1 peers { permanent_uuid: "f4f276754af04e6e9c745ce1ad4eaa94" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 43461 } } }
I20260810 18:50:37.624571 2604 tablet_replica.cc:442] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:50:37.625658 2607 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:50:37.626060 2607 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260810 18:50:37.626367 2351 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260810 18:50:37.630348 2351 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260810 18:50:37.679582 2501 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260810 18:50:37.679960 2501 txn_status_manager.cc:243] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260810 18:50:37.680320 2501 txn_status_manager.cc:205] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20260810 18:50:37.688572 2501 txn_status_manager.cc:336] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260810 18:50:37.723996 2501 txn_status_manager.cc:243] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260810 18:50:37.724273 2501 txn_status_manager.cc:205] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20260810 18:50:37.730657 2501 txn_status_manager.cc:336] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20260810 18:50:37.742169 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:58936
W20260810 18:50:37.744081 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:58936
W20260810 18:50:37.745702 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:58936
W20260810 18:50:37.747372 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:58936
W20260810 18:50:37.749110 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:58936
W20260810 18:50:37.750721 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:58936
W20260810 18:50:37.761013 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:58948
W20260810 18:50:37.762605 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:58948
W20260810 18:50:37.764300 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:58948
W20260810 18:50:37.765831 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:58948
W20260810 18:50:37.767369 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:58948
W20260810 18:50:37.768945 2527 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:58948
I20260810 18:50:37.772949 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:37.788178 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:37.788872 29046 tablet_replica.cc:330] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94: stopping tablet replica
I20260810 18:50:37.789534 29046 raft_consensus.cc:2229] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:37.789985 29046 raft_consensus.cc:2258] T 11111111111111111111111111111111 P f4f276754af04e6e9c745ce1ad4eaa94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:37.809911 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:37.819711 29046 master.cc:537] Master@127.28.93.190:41787 shutting down...
I20260810 18:50:37.833618 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:37.834072 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:37.834438 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P 439bc00bf2c443d984d2d56ddfe89aa7: stopping tablet replica
I20260810 18:50:37.853010 29046 master.cc:552] Master@127.28.93.190:41787 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (625 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20260810 18:50:37.874612 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:38429
I20260810 18:50:37.875588 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:37.880158 2619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:37.880950 2620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:37.883299 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:37.883539 2623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:37.884343 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:37.884523 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:37.884670 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387837884656 us; error 0 us; skew 500 ppm
I20260810 18:50:37.885172 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:37.887270 29046 webserver.cc:460] Webserver started at http://127.28.93.190:43415/ using document root <none> and password file <none>
I20260810 18:50:37.887748 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:37.887917 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:37.888170 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:37.889187 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "ba246cd7920c412489936d21d2714488"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-nj21"
I20260810 18:50:37.893566 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:50:37.896593 2632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:37.897270 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:37.897521 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "ba246cd7920c412489936d21d2714488"
format_stamp: "Formatted at 2026-08-10 18:50:37 on dist-test-slave-nj21"
I20260810 18:50:37.897781 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:37.909310 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:37.910187 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:37.943995 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:38429
I20260810 18:50:37.944080 2725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:38429 every 8 connection(s)
I20260810 18:50:37.947520 2727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:37.957177 2727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488: Bootstrap starting.
I20260810 18:50:37.961145 2727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:37.964737 2727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488: No bootstrap required, opened a new log
I20260810 18:50:37.966476 2727 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba246cd7920c412489936d21d2714488" member_type: VOTER }
I20260810 18:50:37.966837 2727 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:37.967010 2727 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba246cd7920c412489936d21d2714488, State: Initialized, Role: FOLLOWER
I20260810 18:50:37.967473 2727 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba246cd7920c412489936d21d2714488" member_type: VOTER }
I20260810 18:50:37.967911 2727 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:37.968094 2727 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:37.968290 2727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:37.972343 2727 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba246cd7920c412489936d21d2714488" member_type: VOTER }
I20260810 18:50:37.972841 2727 leader_election.cc:302] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba246cd7920c412489936d21d2714488; no voters:
I20260810 18:50:37.973882 2727 leader_election.cc:288] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:37.974164 2731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:37.975386 2731 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 1 LEADER]: Becoming Leader. State: Replica: ba246cd7920c412489936d21d2714488, State: Running, Role: LEADER
I20260810 18:50:37.976034 2731 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba246cd7920c412489936d21d2714488" member_type: VOTER }
I20260810 18:50:37.976538 2727 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:37.978583 2736 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba246cd7920c412489936d21d2714488. Latest consensus state: current_term: 1 leader_uuid: "ba246cd7920c412489936d21d2714488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba246cd7920c412489936d21d2714488" member_type: VOTER } }
I20260810 18:50:37.978663 2732 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba246cd7920c412489936d21d2714488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba246cd7920c412489936d21d2714488" member_type: VOTER } }
I20260810 18:50:37.979272 2736 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:37.979308 2732 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:37.987403 2741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:37.991955 2741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:37.992972 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:37.999831 2741 catalog_manager.cc:1347] Generated new cluster ID: 1b23c9c712a248bba45daf0c3638c725
I20260810 18:50:38.000088 2741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:38.024102 2741 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:38.025730 2741 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:38.036795 2741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488: Generated new TSK 0
I20260810 18:50:38.037432 2741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:38.059119 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:38.064618 2766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:38.065582 2768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:38.067122 2771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:38.067329 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:38.068202 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:38.068379 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:38.068495 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387838068486 us; error 0 us; skew 500 ppm
I20260810 18:50:38.068948 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:38.071166 29046 webserver.cc:460] Webserver started at http://127.28.93.129:38381/ using document root <none> and password file <none>
I20260810 18:50:38.071661 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:38.071900 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:38.072144 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:38.073284 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "4cbcc3bd46e3461ea29137356ce4231f"
format_stamp: "Formatted at 2026-08-10 18:50:38 on dist-test-slave-nj21"
I20260810 18:50:38.077733 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 18:50:38.080907 2776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:38.081585 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:38.081853 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "4cbcc3bd46e3461ea29137356ce4231f"
format_stamp: "Formatted at 2026-08-10 18:50:38 on dist-test-slave-nj21"
I20260810 18:50:38.082125 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:38.109858 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:38.110816 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:38.115711 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:38.116029 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 18:50:38.116343 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:38.116544 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:38.165549 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:38887
I20260810 18:50:38.165639 2896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:38887 every 8 connection(s)
I20260810 18:50:38.181293 2897 heartbeater.cc:347] Connected to a master server at 127.28.93.190:38429
I20260810 18:50:38.182664 2897 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:38.185640 2897 heartbeater.cc:511] Master 127.28.93.190:38429 requested a full tablet report, sending...
I20260810 18:50:38.193830 2667 ts_manager.cc:194] Registered new tserver with Master: 4cbcc3bd46e3461ea29137356ce4231f (127.28.93.129:38887)
I20260810 18:50:38.194078 29046 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.018303232s
I20260810 18:50:38.199129 2667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47794
I20260810 18:50:38.282960 2843 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260810 18:50:38.288764 2843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:38.331399 2915 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f: Bootstrap starting.
I20260810 18:50:38.345064 2915 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.353585 2915 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f: No bootstrap required, opened a new log
I20260810 18:50:38.354784 2915 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.000s
I20260810 18:50:38.359789 2915 raft_consensus.cc:348] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "4cbcc3bd46e3461ea29137356ce4231f" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 38887 } }
I20260810 18:50:38.360445 2915 raft_consensus.cc:374] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.360909 2915 raft_consensus.cc:729] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbcc3bd46e3461ea29137356ce4231f, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.361682 2915 consensus_queue.cc:260] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "4cbcc3bd46e3461ea29137356ce4231f" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 38887 } }
I20260810 18:50:38.363206 2915 raft_consensus.cc:388] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:38.364064 2915 raft_consensus.cc:482] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:38.364567 2915 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.379957 2915 raft_consensus.cc:504] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "4cbcc3bd46e3461ea29137356ce4231f" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 38887 } }
I20260810 18:50:38.381433 2915 leader_election.cc:302] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cbcc3bd46e3461ea29137356ce4231f; no voters:
I20260810 18:50:38.384899 2915 leader_election.cc:288] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:38.385857 2920 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.389518 2920 raft_consensus.cc:686] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 1 LEADER]: Becoming Leader. State: Replica: 4cbcc3bd46e3461ea29137356ce4231f, State: Running, Role: LEADER
I20260810 18:50:38.389827 2897 heartbeater.cc:503] Master 127.28.93.190:38429 was elected leader, sending a full tablet report...
I20260810 18:50:38.390825 2920 consensus_queue.cc:237] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "4cbcc3bd46e3461ea29137356ce4231f" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 38887 } }
I20260810 18:50:38.392599 2915 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f: Time spent starting tablet: real 0.037s user 0.030s sys 0.008s
I20260810 18:50:38.399907 2921 tablet_replica.cc:439] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cbcc3bd46e3461ea29137356ce4231f" committed_config { opid_index: -1 peers { permanent_uuid: "4cbcc3bd46e3461ea29137356ce4231f" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 38887 } } }
I20260810 18:50:38.402168 2921 tablet_replica.cc:442] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:50:38.403517 2922 tablet_replica.cc:439] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f: TxnStatusTablet state changed. Reason: New leader 4cbcc3bd46e3461ea29137356ce4231f. Latest consensus state: current_term: 1 leader_uuid: "4cbcc3bd46e3461ea29137356ce4231f" committed_config { opid_index: -1 peers { permanent_uuid: "4cbcc3bd46e3461ea29137356ce4231f" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 38887 } } }
I20260810 18:50:38.405596 2922 tablet_replica.cc:442] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f: This TxnStatusTablet replica's current role is: LEADER
W20260810 18:50:38.411698 2667 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:38.414565 2927 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:50:38.416316 2927 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260810 18:50:38.424705 2667 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260810 18:50:38.511562 2843 tablet_service.cc:1467] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260810 18:50:38.515904 2843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:38.567030 2915 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f: Bootstrap starting.
I20260810 18:50:38.586521 2915 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:38.602067 2915 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f: No bootstrap required, opened a new log
I20260810 18:50:38.604193 2915 ts_tablet_manager.cc:1397] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f: Time spent bootstrapping tablet: real 0.038s user 0.035s sys 0.000s
I20260810 18:50:38.614339 2915 raft_consensus.cc:348] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "4cbcc3bd46e3461ea29137356ce4231f" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 38887 } }
I20260810 18:50:38.616916 2915 raft_consensus.cc:374] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:38.618083 2915 raft_consensus.cc:729] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbcc3bd46e3461ea29137356ce4231f, State: Initialized, Role: FOLLOWER
I20260810 18:50:38.621049 2915 consensus_queue.cc:260] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "4cbcc3bd46e3461ea29137356ce4231f" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 38887 } }
I20260810 18:50:38.623142 2915 raft_consensus.cc:388] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:38.624053 2915 raft_consensus.cc:482] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:38.625361 2915 raft_consensus.cc:3037] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:38.645709 2915 raft_consensus.cc:504] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "4cbcc3bd46e3461ea29137356ce4231f" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 38887 } }
I20260810 18:50:38.649057 2915 leader_election.cc:302] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cbcc3bd46e3461ea29137356ce4231f; no voters:
I20260810 18:50:38.652537 2915 leader_election.cc:288] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:38.653322 2920 raft_consensus.cc:2781] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:38.658475 2920 raft_consensus.cc:686] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 1 LEADER]: Becoming Leader. State: Replica: 4cbcc3bd46e3461ea29137356ce4231f, State: Running, Role: LEADER
I20260810 18:50:38.663890 2915 ts_tablet_manager.cc:1428] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f: Time spent starting tablet: real 0.058s user 0.048s sys 0.008s
I20260810 18:50:38.663275 2920 consensus_queue.cc:237] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "4cbcc3bd46e3461ea29137356ce4231f" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 38887 } }
W20260810 18:50:38.681066 2899 tablet.cc:2369] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:50:38.693794 2667 catalog_manager.cc:5328] Ignoring report from unknown tablet 22222222222222222222222222222222
W20260810 18:50:38.717198 2667 catalog_manager.cc:5328] Ignoring report from unknown tablet 22222222222222222222222222222222
I20260810 18:50:38.785431 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:38.874886 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:38.875751 29046 tablet_replica.cc:330] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f: stopping tablet replica
I20260810 18:50:38.876399 29046 raft_consensus.cc:2229] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:38.876880 29046 raft_consensus.cc:2258] T 22222222222222222222222222222222 P 4cbcc3bd46e3461ea29137356ce4231f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:38.879437 29046 tablet_replica.cc:330] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f: stopping tablet replica
I20260810 18:50:38.879930 29046 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:38.880339 29046 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 4cbcc3bd46e3461ea29137356ce4231f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:38.892247 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:38.903944 29046 master.cc:537] Master@127.28.93.190:38429 shutting down...
I20260810 18:50:38.918421 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:38.918968 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:38.919279 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P ba246cd7920c412489936d21d2714488: stopping tablet replica
I20260810 18:50:38.936275 29046 master.cc:552] Master@127.28.93.190:38429 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (1086 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (7429 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20260810 18:50:38.961467 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:37621
I20260810 18:50:38.962474 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:38.967249 2936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:38.968767 2938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:38.969712 2942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:38.970335 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:38.970988 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:38.971167 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:38.971288 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387838971278 us; error 0 us; skew 500 ppm
I20260810 18:50:38.971853 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:38.973954 29046 webserver.cc:460] Webserver started at http://127.28.93.190:39647/ using document root <none> and password file <none>
I20260810 18:50:38.974359 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:38.974500 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:38.974685 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:38.975709 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "175031a41ac34524aeb52819a7d880e9"
format_stamp: "Formatted at 2026-08-10 18:50:38 on dist-test-slave-nj21"
I20260810 18:50:38.979976 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:50:38.983144 2955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:38.983860 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:38.984125 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "175031a41ac34524aeb52819a7d880e9"
format_stamp: "Formatted at 2026-08-10 18:50:38 on dist-test-slave-nj21"
I20260810 18:50:38.984406 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:38.993999 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:38.994988 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:39.033428 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:37621
I20260810 18:50:39.033541 3039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:37621 every 8 connection(s)
I20260810 18:50:39.037158 3041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:39.047801 3041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9: Bootstrap starting.
I20260810 18:50:39.052299 3041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:39.056391 3041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9: No bootstrap required, opened a new log
I20260810 18:50:39.058357 3041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175031a41ac34524aeb52819a7d880e9" member_type: VOTER }
I20260810 18:50:39.058729 3041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:39.058915 3041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 175031a41ac34524aeb52819a7d880e9, State: Initialized, Role: FOLLOWER
I20260810 18:50:39.059523 3041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175031a41ac34524aeb52819a7d880e9" member_type: VOTER }
I20260810 18:50:39.060021 3041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:39.060222 3041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:39.060462 3041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:39.064771 3041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175031a41ac34524aeb52819a7d880e9" member_type: VOTER }
I20260810 18:50:39.065248 3041 leader_election.cc:302] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 175031a41ac34524aeb52819a7d880e9; no voters:
I20260810 18:50:39.066375 3041 leader_election.cc:288] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:39.066684 3048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:39.068019 3048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 1 LEADER]: Becoming Leader. State: Replica: 175031a41ac34524aeb52819a7d880e9, State: Running, Role: LEADER
I20260810 18:50:39.068670 3048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175031a41ac34524aeb52819a7d880e9" member_type: VOTER }
I20260810 18:50:39.069269 3041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:39.071424 3049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "175031a41ac34524aeb52819a7d880e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175031a41ac34524aeb52819a7d880e9" member_type: VOTER } }
I20260810 18:50:39.071409 3051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 175031a41ac34524aeb52819a7d880e9. Latest consensus state: current_term: 1 leader_uuid: "175031a41ac34524aeb52819a7d880e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "175031a41ac34524aeb52819a7d880e9" member_type: VOTER } }
I20260810 18:50:39.072157 3049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:39.072235 3051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:39.077613 3055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:39.082367 3055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:39.085005 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:39.090965 3055 catalog_manager.cc:1347] Generated new cluster ID: e2e7fab109d34836b61ae8efe54ea199
I20260810 18:50:39.091198 3055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:39.132149 3055 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:39.133898 3055 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:39.152033 3055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9: Generated new TSK 0
I20260810 18:50:39.152690 3055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:39.216466 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:39.222251 3076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:39.223539 3077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:39.225494 3079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:39.226079 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:39.226850 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:39.227027 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:39.227165 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387839227152 us; error 0 us; skew 500 ppm
I20260810 18:50:39.227694 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:39.229871 29046 webserver.cc:460] Webserver started at http://127.28.93.129:36025/ using document root <none> and password file <none>
I20260810 18:50:39.230341 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:39.230507 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:39.230746 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:39.231875 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "7b240ea59d7e4dc6a479a1bad424084c"
format_stamp: "Formatted at 2026-08-10 18:50:39 on dist-test-slave-nj21"
I20260810 18:50:39.236238 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:50:39.239291 3085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:39.240018 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:39.240265 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "7b240ea59d7e4dc6a479a1bad424084c"
format_stamp: "Formatted at 2026-08-10 18:50:39 on dist-test-slave-nj21"
I20260810 18:50:39.240537 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:39.261355 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:39.262343 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:39.263872 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:39.266175 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:39.266366 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:39.266574 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:39.266716 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:39.308039 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:45957
I20260810 18:50:39.308167 3185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:45957 every 8 connection(s)
I20260810 18:50:39.312577 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:39.320012 3194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:39.321178 3195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:39.324378 3197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:39.325344 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:39.326438 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:39.326678 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:39.326848 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387839326832 us; error 0 us; skew 500 ppm
I20260810 18:50:39.327353 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:39.327607 3186 heartbeater.cc:347] Connected to a master server at 127.28.93.190:37621
I20260810 18:50:39.328019 3186 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:39.328929 3186 heartbeater.cc:511] Master 127.28.93.190:37621 requested a full tablet report, sending...
I20260810 18:50:39.330075 29046 webserver.cc:460] Webserver started at http://127.28.93.130:45891/ using document root <none> and password file <none>
I20260810 18:50:39.330569 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:39.330751 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:39.331003 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:39.331092 2976 ts_manager.cc:194] Registered new tserver with Master: 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129:45957)
I20260810 18:50:39.332190 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-1-root/instance:
uuid: "6f48eddf24e243d589ca90dae55fa580"
format_stamp: "Formatted at 2026-08-10 18:50:39 on dist-test-slave-nj21"
I20260810 18:50:39.333060 2976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47258
I20260810 18:50:39.337195 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:50:39.340328 3202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:39.341073 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:39.341344 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-1-root
uuid: "6f48eddf24e243d589ca90dae55fa580"
format_stamp: "Formatted at 2026-08-10 18:50:39 on dist-test-slave-nj21"
I20260810 18:50:39.341603 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:39.373358 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:39.374501 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:39.375937 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:39.378155 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:39.378336 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:39.378547 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:39.378684 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:39.421123 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.130:37273
I20260810 18:50:39.421270 3314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.130:37273 every 8 connection(s)
I20260810 18:50:39.425835 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:39.433146 3319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:39.435494 3323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:39.437650 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:39.437613 3326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:39.439055 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:39.439316 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:39.439514 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387839439498 us; error 0 us; skew 500 ppm
I20260810 18:50:39.440042 3316 heartbeater.cc:347] Connected to a master server at 127.28.93.190:37621
I20260810 18:50:39.440161 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:39.440393 3316 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:39.441069 3316 heartbeater.cc:511] Master 127.28.93.190:37621 requested a full tablet report, sending...
I20260810 18:50:39.442919 29046 webserver.cc:460] Webserver started at http://127.28.93.131:40179/ using document root <none> and password file <none>
I20260810 18:50:39.443308 2976 ts_manager.cc:194] Registered new tserver with Master: 6f48eddf24e243d589ca90dae55fa580 (127.28.93.130:37273)
I20260810 18:50:39.443588 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:39.443887 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:39.444209 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:39.445612 2976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47266
I20260810 18:50:39.445770 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-2-root/instance:
uuid: "e54624f937ef44a8883692477b52416f"
format_stamp: "Formatted at 2026-08-10 18:50:39 on dist-test-slave-nj21"
I20260810 18:50:39.451141 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 18:50:39.454212 3336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:39.454931 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:50:39.455188 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-2-root
uuid: "e54624f937ef44a8883692477b52416f"
format_stamp: "Formatted at 2026-08-10 18:50:39 on dist-test-slave-nj21"
I20260810 18:50:39.455436 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:39.468997 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:39.470068 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:39.471396 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:39.473552 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:39.473735 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:39.473953 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:39.474095 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:39.518972 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.131:46615
I20260810 18:50:39.519088 3449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.131:46615 every 8 connection(s)
I20260810 18:50:39.534267 3450 heartbeater.cc:347] Connected to a master server at 127.28.93.190:37621
I20260810 18:50:39.534670 3450 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:39.535323 3450 heartbeater.cc:511] Master 127.28.93.190:37621 requested a full tablet report, sending...
I20260810 18:50:39.537101 2976 ts_manager.cc:194] Registered new tserver with Master: e54624f937ef44a8883692477b52416f (127.28.93.131:46615)
I20260810 18:50:39.538033 29046 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015779913s
I20260810 18:50:39.539243 2976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47270
I20260810 18:50:39.561125 29046 test_util.cc:268] Using random seed: -727994991
I20260810 18:50:39.584913 2976 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47286:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:50:39.634109 3132 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae8c88525f74ebc994c2bbffc3c35f5 (DEFAULT_TABLE table=test-workload [id=8a8133c0676b4d848f3a083faf7ebcf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:39.635537 3132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae8c88525f74ebc994c2bbffc3c35f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:39.638478 3245 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae8c88525f74ebc994c2bbffc3c35f5 (DEFAULT_TABLE table=test-workload [id=8a8133c0676b4d848f3a083faf7ebcf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:39.639847 3245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae8c88525f74ebc994c2bbffc3c35f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:39.641938 3389 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae8c88525f74ebc994c2bbffc3c35f5 (DEFAULT_TABLE table=test-workload [id=8a8133c0676b4d848f3a083faf7ebcf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:39.643335 3389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae8c88525f74ebc994c2bbffc3c35f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:39.653474 3484 tablet_bootstrap.cc:492] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Bootstrap starting.
I20260810 18:50:39.656884 3485 tablet_bootstrap.cc:492] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580: Bootstrap starting.
I20260810 18:50:39.662235 3484 tablet_bootstrap.cc:654] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:39.662585 3485 tablet_bootstrap.cc:654] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:39.664989 3486 tablet_bootstrap.cc:492] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f: Bootstrap starting.
I20260810 18:50:39.667652 3484 tablet_bootstrap.cc:492] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: No bootstrap required, opened a new log
I20260810 18:50:39.668195 3484 ts_tablet_manager.cc:1397] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260810 18:50:39.670169 3485 tablet_bootstrap.cc:492] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580: No bootstrap required, opened a new log
I20260810 18:50:39.670671 3485 ts_tablet_manager.cc:1397] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260810 18:50:39.670567 3484 raft_consensus.cc:348] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } }
I20260810 18:50:39.671305 3484 raft_consensus.cc:374] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:39.671564 3484 raft_consensus.cc:729] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b240ea59d7e4dc6a479a1bad424084c, State: Initialized, Role: FOLLOWER
I20260810 18:50:39.672380 3484 consensus_queue.cc:260] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } }
I20260810 18:50:39.673081 3486 tablet_bootstrap.cc:654] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:39.673739 3485 raft_consensus.cc:348] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } }
I20260810 18:50:39.675913 3186 heartbeater.cc:503] Master 127.28.93.190:37621 was elected leader, sending a full tablet report...
I20260810 18:50:39.676139 3484 ts_tablet_manager.cc:1428] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 18:50:39.674551 3485 raft_consensus.cc:374] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:39.679502 3485 raft_consensus.cc:729] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f48eddf24e243d589ca90dae55fa580, State: Initialized, Role: FOLLOWER
I20260810 18:50:39.680334 3485 consensus_queue.cc:260] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } }
I20260810 18:50:39.683624 3485 ts_tablet_manager.cc:1428] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260810 18:50:39.685074 3486 tablet_bootstrap.cc:492] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f: No bootstrap required, opened a new log
I20260810 18:50:39.685557 3486 ts_tablet_manager.cc:1397] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20260810 18:50:39.685526 3316 heartbeater.cc:503] Master 127.28.93.190:37621 was elected leader, sending a full tablet report...
I20260810 18:50:39.688714 3486 raft_consensus.cc:348] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } }
I20260810 18:50:39.689489 3486 raft_consensus.cc:374] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:39.689728 3486 raft_consensus.cc:729] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e54624f937ef44a8883692477b52416f, State: Initialized, Role: FOLLOWER
I20260810 18:50:39.690322 3486 consensus_queue.cc:260] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } }
I20260810 18:50:39.692723 3450 heartbeater.cc:503] Master 127.28.93.190:37621 was elected leader, sending a full tablet report...
I20260810 18:50:39.693145 3486 ts_tablet_manager.cc:1428] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 18:50:39.897204 3496 raft_consensus.cc:482] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:39.897680 3496 raft_consensus.cc:504] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } }
I20260810 18:50:39.899766 3496 leader_election.cc:288] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129:45957), 6f48eddf24e243d589ca90dae55fa580 (127.28.93.130:37273)
I20260810 18:50:39.911201 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae8c88525f74ebc994c2bbffc3c35f5" candidate_uuid: "e54624f937ef44a8883692477b52416f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b240ea59d7e4dc6a479a1bad424084c" is_pre_election: true
I20260810 18:50:39.912083 3149 raft_consensus.cc:2454] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e54624f937ef44a8883692477b52416f in term 0.
I20260810 18:50:39.912282 3260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae8c88525f74ebc994c2bbffc3c35f5" candidate_uuid: "e54624f937ef44a8883692477b52416f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f48eddf24e243d589ca90dae55fa580" is_pre_election: true
I20260810 18:50:39.912935 3260 raft_consensus.cc:2454] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e54624f937ef44a8883692477b52416f in term 0.
I20260810 18:50:39.913245 3340 leader_election.cc:302] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b240ea59d7e4dc6a479a1bad424084c, e54624f937ef44a8883692477b52416f; no voters:
I20260810 18:50:39.914027 3496 raft_consensus.cc:2781] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:39.914315 3496 raft_consensus.cc:482] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:39.914569 3496 raft_consensus.cc:3037] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:39.919560 3496 raft_consensus.cc:504] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } }
I20260810 18:50:39.920984 3496 leader_election.cc:288] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [CANDIDATE]: Term 1 election: Requested vote from peers 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129:45957), 6f48eddf24e243d589ca90dae55fa580 (127.28.93.130:37273)
I20260810 18:50:39.921795 3149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae8c88525f74ebc994c2bbffc3c35f5" candidate_uuid: "e54624f937ef44a8883692477b52416f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b240ea59d7e4dc6a479a1bad424084c"
I20260810 18:50:39.921913 3260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae8c88525f74ebc994c2bbffc3c35f5" candidate_uuid: "e54624f937ef44a8883692477b52416f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f48eddf24e243d589ca90dae55fa580"
I20260810 18:50:39.922402 3149 raft_consensus.cc:3037] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:39.922451 3260 raft_consensus.cc:3037] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:39.929646 3149 raft_consensus.cc:2454] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e54624f937ef44a8883692477b52416f in term 1.
I20260810 18:50:39.929646 3260 raft_consensus.cc:2454] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e54624f937ef44a8883692477b52416f in term 1.
I20260810 18:50:39.930866 3340 leader_election.cc:302] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b240ea59d7e4dc6a479a1bad424084c, e54624f937ef44a8883692477b52416f; no voters:
I20260810 18:50:39.931514 3496 raft_consensus.cc:2781] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:39.932348 3496 raft_consensus.cc:686] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 1 LEADER]: Becoming Leader. State: Replica: e54624f937ef44a8883692477b52416f, State: Running, Role: LEADER
I20260810 18:50:39.933028 3496 consensus_queue.cc:237] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } }
I20260810 18:50:39.940517 2974 catalog_manager.cc:5495] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f reported cstate change: term changed from 0 to 1, leader changed from <none> to e54624f937ef44a8883692477b52416f (127.28.93.131). New cstate: current_term: 1 leader_uuid: "e54624f937ef44a8883692477b52416f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } } }
I20260810 18:50:40.230557 3245 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:40.231802 3132 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:40.236625 3389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:50:40.385406 3496 consensus_queue.cc:1035] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:40.401675 3501 consensus_queue.cc:1035] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
175031a41ac34524aeb52819a7d880e9 | 127.28.93.190:37621 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
6f48eddf24e243d589ca90dae55fa580 | 127.28.93.130:37273 | HEALTHY | <none> | 0 | 0
7b240ea59d7e4dc6a479a1bad424084c | 127.28.93.129:45957 | HEALTHY | <none> | 0 | 0
e54624f937ef44a8883692477b52416f | 127.28.93.131:46615 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 4ae8c88525f74ebc994c2bbffc3c35f5 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129:45957): RUNNING
6f48eddf24e243d589ca90dae55fa580 (127.28.93.130:37273): RUNNING
e54624f937ef44a8883692477b52416f (127.28.93.131:46615): RUNNING [LEADER]
All reported replicas are:
A = 7b240ea59d7e4dc6a479a1bad424084c
B = 6f48eddf24e243d589ca90dae55fa580
C = e54624f937ef44a8883692477b52416f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:50:40.459802 29046 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:50:40.811049 3245 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:40.825878 3132 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:40.832623 3389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
175031a41ac34524aeb52819a7d880e9 | 127.28.93.190:37621 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
6f48eddf24e243d589ca90dae55fa580 | 127.28.93.130:37273 | HEALTHY | <none> | 0 | 0
7b240ea59d7e4dc6a479a1bad424084c | 127.28.93.129:45957 | HEALTHY | <none> | 0 | 0
e54624f937ef44a8883692477b52416f | 127.28.93.131:46615 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260810 18:50:40.995491 29046 log_verifier.cc:126] Checking tablet 4ae8c88525f74ebc994c2bbffc3c35f5
I20260810 18:50:41.057085 29046 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 4ae8c88525f74ebc994c2bbffc3c35f5
I20260810 18:50:41.057657 29046 tablet_replica.cc:330] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: stopping tablet replica
I20260810 18:50:41.058166 29046 raft_consensus.cc:2229] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:41.058627 29046 raft_consensus.cc:2258] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:41.260294 3245 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:41.265857 3132 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:41.267827 3389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20260810 18:50:41.418887 3340 consensus_peers.cc:487] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f -> Peer 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129:45957): Couldn't send request to peer 7b240ea59d7e4dc6a479a1bad424084c. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:41.421545 3661 raft_consensus.cc:998] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f: Attempting to remove follower 7b240ea59d7e4dc6a479a1bad424084c from the Raft config. Reason: The tablet replica hosted on peer 7b240ea59d7e4dc6a479a1bad424084c has failed
I20260810 18:50:41.423473 3661 consensus_queue.cc:237] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } }
I20260810 18:50:41.432564 3260 raft_consensus.cc:1261] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 1 FOLLOWER]: Refusing update from remote peer e54624f937ef44a8883692477b52416f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:50:41.434149 3661 consensus_queue.cc:1035] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:41.441098 3664 raft_consensus.cc:2932] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } } }
I20260810 18:50:41.442796 3260 raft_consensus.cc:2932] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } } }
I20260810 18:50:41.451099 2975 catalog_manager.cc:5495] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f reported cstate change: config changed from index -1 to 2, VOTER 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129) evicted. New cstate: current_term: 1 leader_uuid: "e54624f937ef44a8883692477b52416f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
175031a41ac34524aeb52819a7d880e9 | 127.28.93.190:37621 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
6f48eddf24e243d589ca90dae55fa580 | 127.28.93.130:37273 | HEALTHY | <none> | 0 | 0
7b240ea59d7e4dc6a479a1bad424084c | 127.28.93.129:45957 | HEALTHY | <none> | 0 | 0
e54624f937ef44a8883692477b52416f | 127.28.93.131:46615 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
I20260810 18:50:41.465283 3132 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae8c88525f74ebc994c2bbffc3c35f5 with delete_type TABLET_DATA_TOMBSTONED (TS 7b240ea59d7e4dc6a479a1bad424084c not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:40376
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 4ae8c88525f74ebc994c2bbffc3c35f5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129:45957): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
6f48eddf24e243d589ca90dae55fa580 (127.28.93.130:37273): RUNNING
e54624f937ef44a8883692477b52416f (127.28.93.131:46615): RUNNING [LEADER]
All reported replicas are:
A = 7b240ea59d7e4dc6a479a1bad424084c
B = 6f48eddf24e243d589ca90dae55fa580
C = e54624f937ef44a8883692477b52416f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C* | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:50:41.468251 3406 consensus_queue.cc:237] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } } peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } }
I20260810 18:50:41.472668 3674 ts_tablet_manager.cc:1905] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260810 18:50:41.474524 3340 consensus_peers.cc:487] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f -> Peer 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129:45957): Couldn't send request to peer 7b240ea59d7e4dc6a479a1bad424084c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:41.474628 3260 raft_consensus.cc:1261] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 1 FOLLOWER]: Refusing update from remote peer e54624f937ef44a8883692477b52416f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260810 18:50:41.476985 29046 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:50:41.475957 3661 consensus_queue.cc:1035] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:41.482473 3664 raft_consensus.cc:2932] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } } peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } }
I20260810 18:50:41.483749 3260 raft_consensus.cc:2932] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } } peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } }
I20260810 18:50:41.487841 3674 ts_tablet_manager.cc:1918] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260810 18:50:41.488245 3674 log.cc:1196] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Deleting WAL directory at /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786387803638833-29046-0/minicluster-data/ts-0-root/wals/4ae8c88525f74ebc994c2bbffc3c35f5
I20260810 18:50:41.490249 2961 catalog_manager.cc:4841] TS 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129:45957): tablet 4ae8c88525f74ebc994c2bbffc3c35f5 (table test-workload [id=8a8133c0676b4d848f3a083faf7ebcf4]) successfully deleted
I20260810 18:50:41.490892 2960 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4ae8c88525f74ebc994c2bbffc3c35f5 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260810 18:50:41.493479 2975 catalog_manager.cc:5495] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 reported cstate change: config changed from index 2 to 3, VOTER 7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129) added. New cstate: current_term: 1 leader_uuid: "e54624f937ef44a8883692477b52416f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } } peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } } }
I20260810 18:50:41.830842 3245 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:41.844168 3389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:50:41.844328 3132 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
175031a41ac34524aeb52819a7d880e9 | 127.28.93.190:37621 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
6f48eddf24e243d589ca90dae55fa580 | 127.28.93.130:37273 | HEALTHY | <none> | 0 | 0
7b240ea59d7e4dc6a479a1bad424084c | 127.28.93.129:45957 | HEALTHY | <none> | 0 | 0
e54624f937ef44a8883692477b52416f | 127.28.93.131:46615 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 4ae8c88525f74ebc994c2bbffc3c35f5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
6f48eddf24e243d589ca90dae55fa580 (127.28.93.130:37273): RUNNING
e54624f937ef44a8883692477b52416f (127.28.93.131:46615): RUNNING [LEADER]
7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129:45957): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 6f48eddf24e243d589ca90dae55fa580
B = e54624f937ef44a8883692477b52416f
C = 7b240ea59d7e4dc6a479a1bad424084c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 3 | Yes
B | A B* C | 1 | 3 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:50:42.015949 29046 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:50:42.065097 3725 ts_tablet_manager.cc:927] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Initiating tablet copy from peer e54624f937ef44a8883692477b52416f (127.28.93.131:46615)
I20260810 18:50:42.066522 3725 tablet_copy_client.cc:316] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: tablet copy: Beginning tablet copy session from remote peer at address 127.28.93.131:46615
I20260810 18:50:42.077243 3424 tablet_copy_service.cc:140] P e54624f937ef44a8883692477b52416f: Received BeginTabletCopySession request for tablet 4ae8c88525f74ebc994c2bbffc3c35f5 from peer 7b240ea59d7e4dc6a479a1bad424084c ({username='slave'} at 127.0.0.1:34812)
I20260810 18:50:42.077695 3424 tablet_copy_service.cc:161] P e54624f937ef44a8883692477b52416f: Beginning new tablet copy session on tablet 4ae8c88525f74ebc994c2bbffc3c35f5 from peer 7b240ea59d7e4dc6a479a1bad424084c at {username='slave'} at 127.0.0.1:34812: session id = 7b240ea59d7e4dc6a479a1bad424084c-4ae8c88525f74ebc994c2bbffc3c35f5
I20260810 18:50:42.083082 3424 tablet_copy_source_session.cc:205] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:50:42.085050 3725 ts_tablet_manager.cc:1905] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260810 18:50:42.094631 3725 ts_tablet_manager.cc:1918] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260810 18:50:42.095153 3725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae8c88525f74ebc994c2bbffc3c35f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:42.099764 3725 tablet_copy_client.cc:791] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: tablet copy: Starting download of 0 data blocks...
I20260810 18:50:42.100203 3725 tablet_copy_client.cc:655] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: tablet copy: Starting download of 1 WAL segments...
I20260810 18:50:42.103027 3725 tablet_copy_client.cc:531] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:50:42.108310 3725 tablet_bootstrap.cc:492] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Bootstrap starting.
I20260810 18:50:42.124238 3725 tablet_bootstrap.cc:492] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:50:42.124843 3725 tablet_bootstrap.cc:492] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Bootstrap complete.
I20260810 18:50:42.125294 3725 ts_tablet_manager.cc:1397] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.001s
I20260810 18:50:42.127194 3725 raft_consensus.cc:348] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } } peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } }
I20260810 18:50:42.127723 3725 raft_consensus.cc:729] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b240ea59d7e4dc6a479a1bad424084c, State: Initialized, Role: FOLLOWER
I20260810 18:50:42.128212 3725 consensus_queue.cc:260] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6f48eddf24e243d589ca90dae55fa580" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 37273 } } peers { permanent_uuid: "e54624f937ef44a8883692477b52416f" member_type: VOTER last_known_addr { host: "127.28.93.131" port: 46615 } } peers { permanent_uuid: "7b240ea59d7e4dc6a479a1bad424084c" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 45957 } }
I20260810 18:50:42.130405 3725 ts_tablet_manager.cc:1428] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:50:42.132037 3424 tablet_copy_service.cc:342] P e54624f937ef44a8883692477b52416f: Request end of tablet copy session 7b240ea59d7e4dc6a479a1bad424084c-4ae8c88525f74ebc994c2bbffc3c35f5 received from {username='slave'} at 127.0.0.1:34812
I20260810 18:50:42.132336 3424 tablet_copy_service.cc:433] P e54624f937ef44a8883692477b52416f: ending tablet copy session 7b240ea59d7e4dc6a479a1bad424084c-4ae8c88525f74ebc994c2bbffc3c35f5 on tablet 4ae8c88525f74ebc994c2bbffc3c35f5 with peer 7b240ea59d7e4dc6a479a1bad424084c
I20260810 18:50:42.459234 3245 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:42.472446 3389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:50:42.477378 3132 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:42.580080 3149 raft_consensus.cc:1203] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
175031a41ac34524aeb52819a7d880e9 | 127.28.93.190:37621 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
6f48eddf24e243d589ca90dae55fa580 | 127.28.93.130:37273 | HEALTHY | <none> | 0 | 0
7b240ea59d7e4dc6a479a1bad424084c | 127.28.93.129:45957 | HEALTHY | <none> | 0 | 0
e54624f937ef44a8883692477b52416f | 127.28.93.131:46615 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 4ae8c88525f74ebc994c2bbffc3c35f5 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
6f48eddf24e243d589ca90dae55fa580 (127.28.93.130:37273): RUNNING
e54624f937ef44a8883692477b52416f (127.28.93.131:46615): RUNNING [LEADER]
7b240ea59d7e4dc6a479a1bad424084c (127.28.93.129:45957): RUNNING
All reported replicas are:
A = 6f48eddf24e243d589ca90dae55fa580
B = e54624f937ef44a8883692477b52416f
C = 7b240ea59d7e4dc6a479a1bad424084c
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 3 | Yes
B | A B* C | 1 | 3 | Yes
C | A B C | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:50:42.666404 29046 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:50:43.223248 3132 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:43.225314 3245 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:43.225353 3389 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
175031a41ac34524aeb52819a7d880e9 | 127.28.93.190:37621 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
6f48eddf24e243d589ca90dae55fa580 | 127.28.93.130:37273 | HEALTHY | <none> | 0 | 0
7b240ea59d7e4dc6a479a1bad424084c | 127.28.93.129:45957 | HEALTHY | <none> | 0 | 0
e54624f937ef44a8883692477b52416f | 127.28.93.131:46615 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260810 18:50:43.417956 29046 log_verifier.cc:126] Checking tablet 4ae8c88525f74ebc994c2bbffc3c35f5
I20260810 18:50:43.473100 29046 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4ae8c88525f74ebc994c2bbffc3c35f5
I20260810 18:50:43.475211 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:43.494362 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:43.494982 29046 tablet_replica.cc:330] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c: stopping tablet replica
I20260810 18:50:43.495563 29046 raft_consensus.cc:2229] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:43.495960 29046 raft_consensus.cc:2258] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 7b240ea59d7e4dc6a479a1bad424084c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:43.514492 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:43.523259 29046 tablet_server.cc:178] TabletServer@127.28.93.130:0 shutting down...
I20260810 18:50:43.540745 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:43.541249 29046 tablet_replica.cc:330] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580: stopping tablet replica
I20260810 18:50:43.541697 29046 raft_consensus.cc:2229] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:43.542150 29046 raft_consensus.cc:2258] T 4ae8c88525f74ebc994c2bbffc3c35f5 P 6f48eddf24e243d589ca90dae55fa580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:43.560935 29046 tablet_server.cc:194] TabletServer@127.28.93.130:0 shutdown complete.
I20260810 18:50:43.569368 29046 tablet_server.cc:178] TabletServer@127.28.93.131:0 shutting down...
I20260810 18:50:43.586114 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:43.586691 29046 tablet_replica.cc:330] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f: stopping tablet replica
I20260810 18:50:43.587182 29046 raft_consensus.cc:2229] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:43.587984 29046 raft_consensus.cc:2258] T 4ae8c88525f74ebc994c2bbffc3c35f5 P e54624f937ef44a8883692477b52416f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:43.608145 29046 tablet_server.cc:194] TabletServer@127.28.93.131:0 shutdown complete.
I20260810 18:50:43.616926 29046 master.cc:537] Master@127.28.93.190:37621 shutting down...
I20260810 18:50:43.631249 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:43.631760 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:43.632057 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P 175031a41ac34524aeb52819a7d880e9: stopping tablet replica
I20260810 18:50:43.649528 29046 master.cc:552] Master@127.28.93.190:37621 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (4712 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20260810 18:50:43.673759 29046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.28.93.190:35255
I20260810 18:50:43.674700 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:43.680397 3828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:43.681283 3827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:43.685304 3836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:43.685427 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:43.686184 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:43.686355 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:43.686515 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387843686501 us; error 0 us; skew 500 ppm
I20260810 18:50:43.687000 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:43.689085 29046 webserver.cc:460] Webserver started at http://127.28.93.190:32791/ using document root <none> and password file <none>
I20260810 18:50:43.689528 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:43.689683 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:43.689925 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:43.690917 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/master-0-root/instance:
uuid: "e566106b7e5e4d64a7abc241e70dbdd8"
format_stamp: "Formatted at 2026-08-10 18:50:43 on dist-test-slave-nj21"
I20260810 18:50:43.694926 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 18:50:43.697983 3847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:43.698618 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:50:43.698868 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/master-0-root
uuid: "e566106b7e5e4d64a7abc241e70dbdd8"
format_stamp: "Formatted at 2026-08-10 18:50:43 on dist-test-slave-nj21"
I20260810 18:50:43.699102 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:43.721139 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:43.722023 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:43.755136 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.190:35255
I20260810 18:50:43.755254 3938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.190:35255 every 8 connection(s)
I20260810 18:50:43.758654 3942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:43.768278 3942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8: Bootstrap starting.
I20260810 18:50:43.772454 3942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:43.776018 3942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8: No bootstrap required, opened a new log
I20260810 18:50:43.777743 3942 raft_consensus.cc:348] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566106b7e5e4d64a7abc241e70dbdd8" member_type: VOTER }
I20260810 18:50:43.778103 3942 raft_consensus.cc:374] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:43.778272 3942 raft_consensus.cc:729] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e566106b7e5e4d64a7abc241e70dbdd8, State: Initialized, Role: FOLLOWER
I20260810 18:50:43.778769 3942 consensus_queue.cc:260] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566106b7e5e4d64a7abc241e70dbdd8" member_type: VOTER }
I20260810 18:50:43.779191 3942 raft_consensus.cc:388] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:50:43.779373 3942 raft_consensus.cc:482] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:50:43.779585 3942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:43.783473 3942 raft_consensus.cc:504] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566106b7e5e4d64a7abc241e70dbdd8" member_type: VOTER }
I20260810 18:50:43.783936 3942 leader_election.cc:302] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e566106b7e5e4d64a7abc241e70dbdd8; no voters:
I20260810 18:50:43.784969 3942 leader_election.cc:288] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:50:43.785226 3949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:43.786592 3949 raft_consensus.cc:686] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 1 LEADER]: Becoming Leader. State: Replica: e566106b7e5e4d64a7abc241e70dbdd8, State: Running, Role: LEADER
I20260810 18:50:43.787200 3949 consensus_queue.cc:237] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566106b7e5e4d64a7abc241e70dbdd8" member_type: VOTER }
I20260810 18:50:43.787832 3942 sys_catalog.cc:564] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:50:43.790359 3950 sys_catalog.cc:455] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e566106b7e5e4d64a7abc241e70dbdd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566106b7e5e4d64a7abc241e70dbdd8" member_type: VOTER } }
I20260810 18:50:43.790485 3954 sys_catalog.cc:455] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e566106b7e5e4d64a7abc241e70dbdd8. Latest consensus state: current_term: 1 leader_uuid: "e566106b7e5e4d64a7abc241e70dbdd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566106b7e5e4d64a7abc241e70dbdd8" member_type: VOTER } }
I20260810 18:50:43.791121 3950 sys_catalog.cc:458] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:43.791190 3954 sys_catalog.cc:458] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [sys.catalog]: This master's current role is: LEADER
I20260810 18:50:43.793953 3960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:50:43.798497 3960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:50:43.803226 29046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260810 18:50:43.806313 3960 catalog_manager.cc:1347] Generated new cluster ID: 0cba60853aa54108ba735b97afceae3b
I20260810 18:50:43.806592 3960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:50:43.820211 3960 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:50:43.821733 3960 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:50:43.835263 3960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8: Generated new TSK 0
I20260810 18:50:43.835847 3960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:50:43.869484 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:43.874943 3981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:43.875823 3982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:43.878254 29046 server_base.cc:1034] running on GCE node
W20260810 18:50:43.878408 3985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:43.879318 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:43.879506 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:43.879622 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387843879614 us; error 0 us; skew 500 ppm
I20260810 18:50:43.880105 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:43.882220 29046 webserver.cc:460] Webserver started at http://127.28.93.129:40431/ using document root <none> and password file <none>
I20260810 18:50:43.882633 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:43.882774 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:43.882963 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:43.884004 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-0-root/instance:
uuid: "fb35c2f473044028b9ff5467e37cf0b3"
format_stamp: "Formatted at 2026-08-10 18:50:43 on dist-test-slave-nj21"
I20260810 18:50:43.888113 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:50:43.891281 3992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:43.891995 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:43.892231 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-0-root
uuid: "fb35c2f473044028b9ff5467e37cf0b3"
format_stamp: "Formatted at 2026-08-10 18:50:43 on dist-test-slave-nj21"
I20260810 18:50:43.892472 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:43.911130 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:43.912178 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:43.913432 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:43.915509 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:43.915692 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:43.915911 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:43.916043 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:43.952313 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.129:44143
I20260810 18:50:43.952448 4102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.129:44143 every 8 connection(s)
I20260810 18:50:43.956555 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:43.963492 4108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:43.963793 4110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:43.967067 4112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:43.967303 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:43.968300 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:43.968554 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:43.968748 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387843968733 us; error 0 us; skew 500 ppm
I20260810 18:50:43.969393 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:43.969959 4103 heartbeater.cc:347] Connected to a master server at 127.28.93.190:35255
I20260810 18:50:43.970355 4103 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:43.971143 4103 heartbeater.cc:511] Master 127.28.93.190:35255 requested a full tablet report, sending...
I20260810 18:50:43.972096 29046 webserver.cc:460] Webserver started at http://127.28.93.130:40359/ using document root <none> and password file <none>
I20260810 18:50:43.972641 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:43.972875 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:43.973183 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:43.973253 3876 ts_manager.cc:194] Registered new tserver with Master: fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143)
I20260810 18:50:43.974642 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-1-root/instance:
uuid: "afa4e0e7bb95422fbff489eb5e8fb531"
format_stamp: "Formatted at 2026-08-10 18:50:43 on dist-test-slave-nj21"
I20260810 18:50:43.975831 3876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41232
I20260810 18:50:43.980054 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 18:50:43.983052 4117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:43.983754 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:43.984007 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-1-root
uuid: "afa4e0e7bb95422fbff489eb5e8fb531"
format_stamp: "Formatted at 2026-08-10 18:50:43 on dist-test-slave-nj21"
I20260810 18:50:43.984259 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:44.005465 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:44.006438 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:44.007745 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:44.009795 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:44.009984 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:44.010192 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:44.010330 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:44.046011 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.130:35695
I20260810 18:50:44.046070 4220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.130:35695 every 8 connection(s)
I20260810 18:50:44.050454 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:44.056515 4232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:44.057097 4233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:44.059959 4236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:44.060266 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:44.061259 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:44.061522 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:44.061704 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387844061690 us; error 0 us; skew 500 ppm
I20260810 18:50:44.062166 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:44.062816 4222 heartbeater.cc:347] Connected to a master server at 127.28.93.190:35255
I20260810 18:50:44.063206 4222 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:44.064018 4222 heartbeater.cc:511] Master 127.28.93.190:35255 requested a full tablet report, sending...
I20260810 18:50:44.064895 29046 webserver.cc:460] Webserver started at http://127.28.93.131:33161/ using document root <none> and password file <none>
I20260810 18:50:44.065430 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:44.065614 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:44.065855 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:44.065871 3876 ts_manager.cc:194] Registered new tserver with Master: afa4e0e7bb95422fbff489eb5e8fb531 (127.28.93.130:35695)
I20260810 18:50:44.066979 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-2-root/instance:
uuid: "a6d283a888024efda27907a232ad6697"
format_stamp: "Formatted at 2026-08-10 18:50:44 on dist-test-slave-nj21"
I20260810 18:50:44.067524 3876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41240
I20260810 18:50:44.071378 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:50:44.074558 4244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:44.075253 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:50:44.075523 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-2-root
uuid: "a6d283a888024efda27907a232ad6697"
format_stamp: "Formatted at 2026-08-10 18:50:44 on dist-test-slave-nj21"
I20260810 18:50:44.075825 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:44.107457 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:44.108527 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:44.109916 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:44.112087 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:44.112354 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:44.112558 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:44.112677 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:44.179273 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.131:44135
I20260810 18:50:44.179391 4357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.131:44135 every 8 connection(s)
I20260810 18:50:44.186174 29046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:50:44.195302 4366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:50:44.196022 4367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:44.202891 4360 heartbeater.cc:347] Connected to a master server at 127.28.93.190:35255
I20260810 18:50:44.203195 29046 server_base.cc:1034] running on GCE node
I20260810 18:50:44.203315 4360 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:44.204281 4360 heartbeater.cc:511] Master 127.28.93.190:35255 requested a full tablet report, sending...
W20260810 18:50:44.204840 4371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:50:44.206039 29046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260810 18:50:44.206262 29046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260810 18:50:44.206444 29046 hybrid_clock.cc:648] HybridClock initialized: now 1786387844206430 us; error 0 us; skew 500 ppm
I20260810 18:50:44.207015 29046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 18:50:44.207049 3876 ts_manager.cc:194] Registered new tserver with Master: a6d283a888024efda27907a232ad6697 (127.28.93.131:44135)
I20260810 18:50:44.209484 3876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41246
I20260810 18:50:44.209959 29046 webserver.cc:460] Webserver started at http://127.28.93.132:34207/ using document root <none> and password file <none>
I20260810 18:50:44.210594 29046 fs_manager.cc:359] Metadata directory not provided
I20260810 18:50:44.210870 29046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:50:44.211205 29046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:50:44.212944 29046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-3-root/instance:
uuid: "37cac165757f48618222a1034a20a40e"
format_stamp: "Formatted at 2026-08-10 18:50:44 on dist-test-slave-nj21"
I20260810 18:50:44.217656 29046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 18:50:44.221040 4379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:44.221758 29046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:50:44.222030 29046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-3-root
uuid: "37cac165757f48618222a1034a20a40e"
format_stamp: "Formatted at 2026-08-10 18:50:44 on dist-test-slave-nj21"
I20260810 18:50:44.222326 29046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:50:44.249738 29046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:50:44.251041 29046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:50:44.252616 29046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:50:44.255100 29046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:50:44.255298 29046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:44.255522 29046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:50:44.255697 29046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:50:44.304454 29046 rpc_server.cc:307] RPC server started. Bound to: 127.28.93.132:42183
I20260810 18:50:44.304629 4494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.93.132:42183 every 8 connection(s)
I20260810 18:50:44.319533 4496 heartbeater.cc:347] Connected to a master server at 127.28.93.190:35255
I20260810 18:50:44.319952 4496 heartbeater.cc:464] Registering TS with master...
I20260810 18:50:44.320645 4496 heartbeater.cc:511] Master 127.28.93.190:35255 requested a full tablet report, sending...
I20260810 18:50:44.322404 3876 ts_manager.cc:194] Registered new tserver with Master: 37cac165757f48618222a1034a20a40e (127.28.93.132:42183)
I20260810 18:50:44.322503 29046 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.014827729s
I20260810 18:50:44.324363 3876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41254
I20260810 18:50:44.345122 29046 test_util.cc:268] Using random seed: -723210997
I20260810 18:50:44.367635 3876 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41266:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:50:44.422494 4165 tablet_service.cc:1467] Processing CreateTablet for tablet 2d53b02ff5a04ae4a84e445a86ab9da0 (DEFAULT_TABLE table=test-workload [id=36277cc1d2144b7da4bc8a507513b5ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:44.424197 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d53b02ff5a04ae4a84e445a86ab9da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:44.426201 4429 tablet_service.cc:1467] Processing CreateTablet for tablet 2d53b02ff5a04ae4a84e445a86ab9da0 (DEFAULT_TABLE table=test-workload [id=36277cc1d2144b7da4bc8a507513b5ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:44.427524 4429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d53b02ff5a04ae4a84e445a86ab9da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:44.429687 4038 tablet_service.cc:1467] Processing CreateTablet for tablet 2d53b02ff5a04ae4a84e445a86ab9da0 (DEFAULT_TABLE table=test-workload [id=36277cc1d2144b7da4bc8a507513b5ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:50:44.431162 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d53b02ff5a04ae4a84e445a86ab9da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:50:44.449373 4541 tablet_bootstrap.cc:492] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e: Bootstrap starting.
I20260810 18:50:44.452346 4548 tablet_bootstrap.cc:492] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3: Bootstrap starting.
I20260810 18:50:44.456327 4547 tablet_bootstrap.cc:492] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531: Bootstrap starting.
I20260810 18:50:44.456739 4541 tablet_bootstrap.cc:654] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:44.460479 4548 tablet_bootstrap.cc:654] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:44.463096 4547 tablet_bootstrap.cc:654] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531: Neither blocks nor log segments found. Creating new log.
I20260810 18:50:44.465513 4541 tablet_bootstrap.cc:492] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e: No bootstrap required, opened a new log
I20260810 18:50:44.465914 4541 ts_tablet_manager.cc:1397] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260810 18:50:44.469005 4548 tablet_bootstrap.cc:492] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3: No bootstrap required, opened a new log
I20260810 18:50:44.468984 4541 raft_consensus.cc:348] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:44.470144 4548 ts_tablet_manager.cc:1397] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.011s
I20260810 18:50:44.470162 4541 raft_consensus.cc:374] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:44.470873 4541 raft_consensus.cc:729] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37cac165757f48618222a1034a20a40e, State: Initialized, Role: FOLLOWER
I20260810 18:50:44.471743 4541 consensus_queue.cc:260] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:44.474869 4548 raft_consensus.cc:348] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:44.475558 4548 raft_consensus.cc:374] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:44.475688 4547 tablet_bootstrap.cc:492] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531: No bootstrap required, opened a new log
I20260810 18:50:44.475944 4548 raft_consensus.cc:729] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb35c2f473044028b9ff5467e37cf0b3, State: Initialized, Role: FOLLOWER
I20260810 18:50:44.476356 4547 ts_tablet_manager.cc:1397] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.004s
I20260810 18:50:44.476866 4548 consensus_queue.cc:260] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:44.479759 4103 heartbeater.cc:503] Master 127.28.93.190:35255 was elected leader, sending a full tablet report...
I20260810 18:50:44.480074 4548 ts_tablet_manager.cc:1428] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260810 18:50:44.480486 4547 raft_consensus.cc:348] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:44.481782 4547 raft_consensus.cc:374] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:50:44.482533 4541 ts_tablet_manager.cc:1428] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
I20260810 18:50:44.482146 4547 raft_consensus.cc:729] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afa4e0e7bb95422fbff489eb5e8fb531, State: Initialized, Role: FOLLOWER
I20260810 18:50:44.483426 4496 heartbeater.cc:503] Master 127.28.93.190:35255 was elected leader, sending a full tablet report...
I20260810 18:50:44.483501 4547 consensus_queue.cc:260] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:44.487941 4222 heartbeater.cc:503] Master 127.28.93.190:35255 was elected leader, sending a full tablet report...
I20260810 18:50:44.489163 4547 ts_tablet_manager.cc:1428] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260810 18:50:44.631646 4553 raft_consensus.cc:482] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:50:44.632179 4553 raft_consensus.cc:504] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:44.634195 4553 leader_election.cc:288] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afa4e0e7bb95422fbff489eb5e8fb531 (127.28.93.130:35695), 37cac165757f48618222a1034a20a40e (127.28.93.132:42183)
I20260810 18:50:44.647786 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d53b02ff5a04ae4a84e445a86ab9da0" candidate_uuid: "fb35c2f473044028b9ff5467e37cf0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" is_pre_election: true
I20260810 18:50:44.648729 4183 raft_consensus.cc:2454] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb35c2f473044028b9ff5467e37cf0b3 in term 0.
I20260810 18:50:44.648663 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d53b02ff5a04ae4a84e445a86ab9da0" candidate_uuid: "fb35c2f473044028b9ff5467e37cf0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37cac165757f48618222a1034a20a40e" is_pre_election: true
I20260810 18:50:44.649466 4448 raft_consensus.cc:2454] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb35c2f473044028b9ff5467e37cf0b3 in term 0.
I20260810 18:50:44.649968 3996 leader_election.cc:302] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afa4e0e7bb95422fbff489eb5e8fb531, fb35c2f473044028b9ff5467e37cf0b3; no voters:
I20260810 18:50:44.650650 4553 raft_consensus.cc:2781] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:50:44.650918 4553 raft_consensus.cc:482] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:50:44.651162 4553 raft_consensus.cc:3037] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:44.656222 4553 raft_consensus.cc:504] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:44.657595 4553 leader_election.cc:288] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [CANDIDATE]: Term 1 election: Requested vote from peers afa4e0e7bb95422fbff489eb5e8fb531 (127.28.93.130:35695), 37cac165757f48618222a1034a20a40e (127.28.93.132:42183)
I20260810 18:50:44.658346 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d53b02ff5a04ae4a84e445a86ab9da0" candidate_uuid: "fb35c2f473044028b9ff5467e37cf0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afa4e0e7bb95422fbff489eb5e8fb531"
I20260810 18:50:44.658502 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d53b02ff5a04ae4a84e445a86ab9da0" candidate_uuid: "fb35c2f473044028b9ff5467e37cf0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37cac165757f48618222a1034a20a40e"
I20260810 18:50:44.658797 4183 raft_consensus.cc:3037] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:44.658989 4448 raft_consensus.cc:3037] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:50:44.663892 4183 raft_consensus.cc:2454] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb35c2f473044028b9ff5467e37cf0b3 in term 1.
I20260810 18:50:44.664260 4448 raft_consensus.cc:2454] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb35c2f473044028b9ff5467e37cf0b3 in term 1.
I20260810 18:50:44.664759 3996 leader_election.cc:302] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afa4e0e7bb95422fbff489eb5e8fb531, fb35c2f473044028b9ff5467e37cf0b3; no voters:
I20260810 18:50:44.665405 4553 raft_consensus.cc:2781] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:50:44.666394 4553 raft_consensus.cc:686] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 1 LEADER]: Becoming Leader. State: Replica: fb35c2f473044028b9ff5467e37cf0b3, State: Running, Role: LEADER
I20260810 18:50:44.667035 4553 consensus_queue.cc:237] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:44.673331 3868 catalog_manager.cc:5495] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129). New cstate: current_term: 1 leader_uuid: "fb35c2f473044028b9ff5467e37cf0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:44.943231 4429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:44.950354 4288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:44.959856 4038 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:50:44.962532 4165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:45.123281 4553 consensus_queue.cc:1035] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:45.173100 4553 consensus_queue.cc:1035] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:50:45.212975 4360 heartbeater.cc:503] Master 127.28.93.190:35255 was elected leader, sending a full tablet report...
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e566106b7e5e4d64a7abc241e70dbdd8 | 127.28.93.190:35255 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
37cac165757f48618222a1034a20a40e | 127.28.93.132:42183 | HEALTHY | <none> | 0 | 0
a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | HEALTHY | <none> | 0 | 0
afa4e0e7bb95422fbff489eb5e8fb531 | 127.28.93.130:35695 | HEALTHY | <none> | 0 | 0
fb35c2f473044028b9ff5467e37cf0b3 | 127.28.93.129:44143 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d53b02ff5a04ae4a84e445a86ab9da0 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
afa4e0e7bb95422fbff489eb5e8fb531 (127.28.93.130:35695): RUNNING
37cac165757f48618222a1034a20a40e (127.28.93.132:42183): RUNNING
fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143): RUNNING [LEADER]
All reported replicas are:
A = afa4e0e7bb95422fbff489eb5e8fb531
B = 37cac165757f48618222a1034a20a40e
C = fb35c2f473044028b9ff5467e37cf0b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:50:45.224109 29046 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:50:45.583420 4429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:45.598724 4038 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:50:45.602022 4288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:45.608708 4165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e566106b7e5e4d64a7abc241e70dbdd8 | 127.28.93.190:35255 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
37cac165757f48618222a1034a20a40e | 127.28.93.132:42183 | HEALTHY | <none> | 0 | 0
a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | HEALTHY | <none> | 0 | 0
afa4e0e7bb95422fbff489eb5e8fb531 | 127.28.93.130:35695 | HEALTHY | <none> | 0 | 0
fb35c2f473044028b9ff5467e37cf0b3 | 127.28.93.129:44143 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260810 18:50:45.819425 29046 log_verifier.cc:126] Checking tablet 2d53b02ff5a04ae4a84e445a86ab9da0
I20260810 18:50:45.874909 29046 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2d53b02ff5a04ae4a84e445a86ab9da0
I20260810 18:50:45.875406 29046 tablet_replica.cc:330] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3: stopping tablet replica
I20260810 18:50:45.875952 29046 raft_consensus.cc:2229] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:45.876821 29046 raft_consensus.cc:2258] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:46.080262 4429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:46.097528 4288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:46.101540 4165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:46.104601 4038 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e566106b7e5e4d64a7abc241e70dbdd8 | 127.28.93.190:35255 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
37cac165757f48618222a1034a20a40e | 127.28.93.132:42183 | HEALTHY | <none> | 0 | 0
a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | HEALTHY | <none> | 0 | 0
afa4e0e7bb95422fbff489eb5e8fb531 | 127.28.93.130:35695 | HEALTHY | <none> | 0 | 0
fb35c2f473044028b9ff5467e37cf0b3 | 127.28.93.129:44143 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d53b02ff5a04ae4a84e445a86ab9da0 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
afa4e0e7bb95422fbff489eb5e8fb531 (127.28.93.130:35695): RUNNING
37cac165757f48618222a1034a20a40e (127.28.93.132:42183): RUNNING
fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = afa4e0e7bb95422fbff489eb5e8fb531
B = 37cac165757f48618222a1034a20a40e
C = fb35c2f473044028b9ff5467e37cf0b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:50:46.302856 29046 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:50:46.655011 4429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:46.662492 4288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:46.670630 4165 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:46.673928 4038 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e566106b7e5e4d64a7abc241e70dbdd8 | 127.28.93.190:35255 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
37cac165757f48618222a1034a20a40e | 127.28.93.132:42183 | HEALTHY | <none> | 0 | 0
a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | HEALTHY | <none> | 0 | 0
afa4e0e7bb95422fbff489eb5e8fb531 | 127.28.93.130:35695 | HEALTHY | <none> | 0 | 0
fb35c2f473044028b9ff5467e37cf0b3 | 127.28.93.129:44143 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d53b02ff5a04ae4a84e445a86ab9da0 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
afa4e0e7bb95422fbff489eb5e8fb531 (127.28.93.130:35695): RUNNING
37cac165757f48618222a1034a20a40e (127.28.93.132:42183): RUNNING
fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = afa4e0e7bb95422fbff489eb5e8fb531
B = 37cac165757f48618222a1034a20a40e
C = fb35c2f473044028b9ff5467e37cf0b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:50:46.883787 29046 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:50:47.092614 4788 raft_consensus.cc:482] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fb35c2f473044028b9ff5467e37cf0b3)
I20260810 18:50:47.093214 4788 raft_consensus.cc:504] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:47.095153 4788 leader_election.cc:288] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers afa4e0e7bb95422fbff489eb5e8fb531 (127.28.93.130:35695), fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143)
I20260810 18:50:47.109586 4183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d53b02ff5a04ae4a84e445a86ab9da0" candidate_uuid: "37cac165757f48618222a1034a20a40e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" is_pre_election: true
I20260810 18:50:47.111073 4057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d53b02ff5a04ae4a84e445a86ab9da0" candidate_uuid: "37cac165757f48618222a1034a20a40e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fb35c2f473044028b9ff5467e37cf0b3" is_pre_election: true
W20260810 18:50:47.112294 4380 leader_election.cc:341] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143): Illegal state: cannot vote while shut down
I20260810 18:50:47.112820 4380 leader_election.cc:302] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37cac165757f48618222a1034a20a40e; no voters: afa4e0e7bb95422fbff489eb5e8fb531, fb35c2f473044028b9ff5467e37cf0b3
I20260810 18:50:47.113665 4788 raft_consensus.cc:2726] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 18:50:47.259358 4824 raft_consensus.cc:482] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fb35c2f473044028b9ff5467e37cf0b3)
I20260810 18:50:47.259835 4824 raft_consensus.cc:504] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:47.261837 4824 leader_election.cc:288] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37cac165757f48618222a1034a20a40e (127.28.93.132:42183), fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143)
I20260810 18:50:47.288412 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d53b02ff5a04ae4a84e445a86ab9da0" candidate_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37cac165757f48618222a1034a20a40e" is_pre_election: true
I20260810 18:50:47.289101 4448 raft_consensus.cc:2454] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afa4e0e7bb95422fbff489eb5e8fb531 in term 1.
I20260810 18:50:47.291347 4125 leader_election.cc:302] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37cac165757f48618222a1034a20a40e, afa4e0e7bb95422fbff489eb5e8fb531; no voters:
I20260810 18:50:47.292176 4824 raft_consensus.cc:2781] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:50:47.292486 4824 raft_consensus.cc:482] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fb35c2f473044028b9ff5467e37cf0b3)
I20260810 18:50:47.292832 4824 raft_consensus.cc:3037] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:50:47.292791 4057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d53b02ff5a04ae4a84e445a86ab9da0" candidate_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fb35c2f473044028b9ff5467e37cf0b3" is_pre_election: true
W20260810 18:50:47.294288 4122 leader_election.cc:341] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143): Illegal state: cannot vote while shut down
I20260810 18:50:47.298563 4824 raft_consensus.cc:504] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:47.300158 4824 leader_election.cc:288] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [CANDIDATE]: Term 2 election: Requested vote from peers 37cac165757f48618222a1034a20a40e (127.28.93.132:42183), fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143)
I20260810 18:50:47.301250 4448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d53b02ff5a04ae4a84e445a86ab9da0" candidate_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "37cac165757f48618222a1034a20a40e"
I20260810 18:50:47.301245 4057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d53b02ff5a04ae4a84e445a86ab9da0" candidate_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fb35c2f473044028b9ff5467e37cf0b3"
I20260810 18:50:47.301975 4448 raft_consensus.cc:3037] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 1 FOLLOWER]: Advancing to term 2
W20260810 18:50:47.302585 4122 leader_election.cc:341] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143): Illegal state: cannot vote while shut down
I20260810 18:50:47.307389 4448 raft_consensus.cc:2454] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afa4e0e7bb95422fbff489eb5e8fb531 in term 2.
I20260810 18:50:47.308329 4125 leader_election.cc:302] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37cac165757f48618222a1034a20a40e, afa4e0e7bb95422fbff489eb5e8fb531; no voters: fb35c2f473044028b9ff5467e37cf0b3
I20260810 18:50:47.309026 4824 raft_consensus.cc:2781] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:50:47.310071 4824 raft_consensus.cc:686] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 2 LEADER]: Becoming Leader. State: Replica: afa4e0e7bb95422fbff489eb5e8fb531, State: Running, Role: LEADER
I20260810 18:50:47.312641 4824 consensus_queue.cc:237] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } }
I20260810 18:50:47.321448 3876 catalog_manager.cc:5495] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 reported cstate change: term changed from 1 to 2, leader changed from fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129) to afa4e0e7bb95422fbff489eb5e8fb531 (127.28.93.130). New cstate: current_term: 2 leader_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb35c2f473044028b9ff5467e37cf0b3" member_type: VOTER last_known_addr { host: "127.28.93.129" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260810 18:50:47.341554 4429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:47.346592 4288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:47.361217 4038 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:47.362555 4165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e566106b7e5e4d64a7abc241e70dbdd8 | 127.28.93.190:35255 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
37cac165757f48618222a1034a20a40e | 127.28.93.132:42183 | HEALTHY | <none> | 0 | 0
a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | HEALTHY | <none> | 0 | 0
afa4e0e7bb95422fbff489eb5e8fb531 | 127.28.93.130:35695 | HEALTHY | <none> | 1 | 0
fb35c2f473044028b9ff5467e37cf0b3 | 127.28.93.129:44143 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 2d53b02ff5a04ae4a84e445a86ab9da0 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
afa4e0e7bb95422fbff489eb5e8fb531 (127.28.93.130:35695): RUNNING
37cac165757f48618222a1034a20a40e (127.28.93.132:42183): RUNNING
fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = afa4e0e7bb95422fbff489eb5e8fb531
B = 37cac165757f48618222a1034a20a40e
C = fb35c2f473044028b9ff5467e37cf0b3
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A* B C | 2 | -1 | Yes
B | A B C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260810 18:50:47.571241 29046 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260810 18:50:47.797245 4122 consensus_queue.cc:579] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [LEADER]: The tablet replica hosted on peer fb35c2f473044028b9ff5467e37cf0b3 has failed
W20260810 18:50:47.797761 4122 consensus_peers.cc:487] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 -> Peer fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143): Couldn't send request to peer fb35c2f473044028b9ff5467e37cf0b3. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:47.845858 4448 raft_consensus.cc:1261] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 2 FOLLOWER]: Refusing update from remote peer afa4e0e7bb95422fbff489eb5e8fb531: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260810 18:50:47.847266 4824 consensus_queue.cc:1035] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260810 18:50:47.863193 4183 consensus_queue.cc:237] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } }
I20260810 18:50:47.866554 4448 raft_consensus.cc:1261] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 2 FOLLOWER]: Refusing update from remote peer afa4e0e7bb95422fbff489eb5e8fb531: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260810 18:50:47.867529 4859 consensus_queue.cc:1035] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260810 18:50:47.872165 4824 raft_consensus.cc:2932] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } }
I20260810 18:50:47.873458 4448 raft_consensus.cc:2932] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } }
I20260810 18:50:47.878830 3850 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2d53b02ff5a04ae4a84e445a86ab9da0 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260810 18:50:47.881825 3876 catalog_manager.cc:5495] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 reported cstate change: config changed from index -1 to 3, VOTER fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129) evicted. New cstate: current_term: 2 leader_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } health_report { overall_health: HEALTHY } } }
I20260810 18:50:47.891870 4038 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d53b02ff5a04ae4a84e445a86ab9da0 with delete_type TABLET_DATA_TOMBSTONED (TS fb35c2f473044028b9ff5467e37cf0b3 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:41374
I20260810 18:50:47.894109 4869 ts_tablet_manager.cc:1905] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260810 18:50:47.895345 4183 consensus_queue.cc:237] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "a6d283a888024efda27907a232ad6697" member_type: NON_VOTER last_known_addr { host: "127.28.93.131" port: 44135 } attrs { promote: true } }
I20260810 18:50:47.900851 4448 raft_consensus.cc:1261] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 2 FOLLOWER]: Refusing update from remote peer afa4e0e7bb95422fbff489eb5e8fb531: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260810 18:50:47.902407 4859 consensus_queue.cc:1035] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260810 18:50:47.909327 4869 ts_tablet_manager.cc:1918] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260810 18:50:47.909818 4869 log.cc:1196] T 2d53b02ff5a04ae4a84e445a86ab9da0 P fb35c2f473044028b9ff5467e37cf0b3: Deleting WAL directory at /tmp/dist-test-taskp7N8DQ/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786387803638833-29046-0/minicluster-data/ts-0-root/wals/2d53b02ff5a04ae4a84e445a86ab9da0
I20260810 18:50:47.909132 4824 raft_consensus.cc:2932] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER a6d283a888024efda27907a232ad6697 (127.28.93.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "a6d283a888024efda27907a232ad6697" member_type: NON_VOTER last_known_addr { host: "127.28.93.131" port: 44135 } attrs { promote: true } } }
I20260810 18:50:47.910843 4448 raft_consensus.cc:2932] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER a6d283a888024efda27907a232ad6697 (127.28.93.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "a6d283a888024efda27907a232ad6697" member_type: NON_VOTER last_known_addr { host: "127.28.93.131" port: 44135 } attrs { promote: true } } }
I20260810 18:50:47.911731 3848 catalog_manager.cc:4841] TS fb35c2f473044028b9ff5467e37cf0b3 (127.28.93.129:44143): tablet 2d53b02ff5a04ae4a84e445a86ab9da0 (table test-workload [id=36277cc1d2144b7da4bc8a507513b5ee]) successfully deleted
I20260810 18:50:47.922266 3876 catalog_manager.cc:5495] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e reported cstate change: config changed from index 3 to 4, NON_VOTER a6d283a888024efda27907a232ad6697 (127.28.93.131) added. New cstate: current_term: 2 leader_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "afa4e0e7bb95422fbff489eb5e8fb531" member_type: VOTER last_known_addr { host: "127.28.93.130" port: 35695 } } peers { permanent_uuid: "37cac165757f48618222a1034a20a40e" member_type: VOTER last_known_addr { host: "127.28.93.132" port: 42183 } } peers { permanent_uuid: "a6d283a888024efda27907a232ad6697" member_type: NON_VOTER last_known_addr { host: "127.28.93.131" port: 44135 } attrs { promote: true } } }
I20260810 18:50:47.925103 3850 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d53b02ff5a04ae4a84e445a86ab9da0 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20260810 18:50:47.934274 4122 consensus_peers.cc:487] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 -> Peer a6d283a888024efda27907a232ad6697 (127.28.93.131:44135): Couldn't send request to peer a6d283a888024efda27907a232ad6697. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d53b02ff5a04ae4a84e445a86ab9da0. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:50:48.145987 4429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:48.150365 4165 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260810 18:50:48.150717 4288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260810 18:50:48.158278 4038 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e566106b7e5e4d64a7abc241e70dbdd8 | 127.28.93.190:35255 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
37cac165757f48618222a1034a20a40e | 127.28.93.132:42183 | HEALTHY | <none> | 0 | 0
a6d283a888024efda27907a232ad6697 | 127.28.93.131:44135 | HEALTHY | <none> | 0 | 0
afa4e0e7bb95422fbff489eb5e8fb531 | 127.28.93.130:35695 | HEALTHY | <none> | 1 | 0
fb35c2f473044028b9ff5467e37cf0b3 | 127.28.93.129:44143 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | fb35c2f473044028b9ff5467e37cf0b3 | 127.28.93.129:44143 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260810 18:50:48.361687 29046 log_verifier.cc:126] Checking tablet 2d53b02ff5a04ae4a84e445a86ab9da0
I20260810 18:50:48.401692 29046 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 2d53b02ff5a04ae4a84e445a86ab9da0
I20260810 18:50:48.405045 29046 tablet_server.cc:178] TabletServer@127.28.93.129:0 shutting down...
I20260810 18:50:48.430392 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:48.447947 29046 tablet_server.cc:194] TabletServer@127.28.93.129:0 shutdown complete.
I20260810 18:50:48.457293 29046 tablet_server.cc:178] TabletServer@127.28.93.130:0 shutting down...
I20260810 18:50:48.468680 4929 ts_tablet_manager.cc:927] T 2d53b02ff5a04ae4a84e445a86ab9da0 P a6d283a888024efda27907a232ad6697: Initiating tablet copy from peer afa4e0e7bb95422fbff489eb5e8fb531 (127.28.93.130:35695)
I20260810 18:50:48.472748 4929 tablet_copy_client.cc:316] T 2d53b02ff5a04ae4a84e445a86ab9da0 P a6d283a888024efda27907a232ad6697: tablet copy: Beginning tablet copy session from remote peer at address 127.28.93.130:35695
I20260810 18:50:48.484686 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:48.485464 29046 tablet_replica.cc:330] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531: stopping tablet replica
I20260810 18:50:48.486145 29046 raft_consensus.cc:2229] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 2 LEADER]: Raft consensus shutting down.
I20260810 18:50:48.487049 29046 raft_consensus.cc:2258] T 2d53b02ff5a04ae4a84e445a86ab9da0 P afa4e0e7bb95422fbff489eb5e8fb531 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260810 18:50:48.492010 4122 proxy.cc:239] Call had error, refreshing address and retrying: Service unavailable: reactor is shutting down (error 108) [suppressed 44 similar messages]
W20260810 18:50:48.499441 4248 connection.cc:441] server connection from 127.0.0.1:53266 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:53266 (request call id 1) could send its response
I20260810 18:50:48.507628 29046 tablet_server.cc:194] TabletServer@127.28.93.130:0 shutdown complete.
I20260810 18:50:48.517160 29046 tablet_server.cc:178] TabletServer@127.28.93.131:0 shutting down...
I20260810 18:50:48.537372 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:48.553516 29046 tablet_server.cc:194] TabletServer@127.28.93.131:0 shutdown complete.
I20260810 18:50:48.561879 29046 tablet_server.cc:178] TabletServer@127.28.93.132:0 shutting down...
I20260810 18:50:48.578743 29046 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260810 18:50:48.579294 29046 tablet_replica.cc:330] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e: stopping tablet replica
I20260810 18:50:48.579836 29046 raft_consensus.cc:2229] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260810 18:50:48.580286 29046 raft_consensus.cc:2258] T 2d53b02ff5a04ae4a84e445a86ab9da0 P 37cac165757f48618222a1034a20a40e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:48.600018 29046 tablet_server.cc:194] TabletServer@127.28.93.132:0 shutdown complete.
I20260810 18:50:48.608881 29046 master.cc:537] Master@127.28.93.190:35255 shutting down...
I20260810 18:50:48.627130 29046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:50:48.627630 29046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:50:48.627970 29046 tablet_replica.cc:330] T 00000000000000000000000000000000 P e566106b7e5e4d64a7abc241e70dbdd8: stopping tablet replica
I20260810 18:50:48.646018 29046 master.cc:552] Master@127.28.93.190:35255 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (4999 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (9712 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (44716 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20260810 18:50:48.679406 29046 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~10 seconds ago
I20260810 18:50:48.679693 29046 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/catalog_manager.cc:4584: suppressed but not reported on 1 messages since previous log ~18 seconds ago
I20260810 18:50:48.679929 29046 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tserver/heartbeater.cc:644: suppressed but not reported on 136 messages since previous log ~35 seconds ago
I20260810 18:50:48.680209 29046 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tablet/tablet.cc:2369: suppressed but not reported on 6 messages since previous log ~9 seconds ago