[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260811 18:02:20.446363 28033 debug-util.cc:398] Leaking SignalData structure 0x7b0800028040 after lost signal to thread 28027
W20260811 18:02:20.448005 28033 kernel_stack_watchdog.cc:198] Thread 28027 stuck at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:637 for 405ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260811 18:02:20.535043 28027 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 1.499s user 0.000s sys 0.004s
W20260811 18:02:20.668131 28027 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 1.633s user 0.006s sys 0.013s
I20260811 18:02:20.673594 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:39985
I20260811 18:02:20.676519 28027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 18:02:20.678808 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:20.709277 28044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:20.711112 28045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:20.729342 28027 server_base.cc:1034] running on GCE node
W20260811 18:02:20.729403 28047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:20.731288 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:20.731734 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:20.731954 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471340731927 us; error 0 us; skew 500 ppm
I20260811 18:02:20.733522 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:20.749650 28027 webserver.cc:460] Webserver started at http://127.27.94.254:38683/ using document root <none> and password file <none>
I20260811 18:02:20.751082 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:20.751376 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:20.752125 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:20.762786 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "6bd094ee89344b97990298d22dd7ef38"
format_stamp: "Formatted at 2026-08-11 18:02:20 on dist-test-slave-kvfs"
I20260811 18:02:20.779672 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.010s sys 0.001s
I20260811 18:02:20.790570 28053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:20.793288 28027 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.004s
I20260811 18:02:20.793865 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "6bd094ee89344b97990298d22dd7ef38"
format_stamp: "Formatted at 2026-08-11 18:02:20 on dist-test-slave-kvfs"
I20260811 18:02:20.794703 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:20.832883 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:20.835402 28027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 18:02:20.836212 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:21.109488 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:39985
I20260811 18:02:21.109721 28137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:39985 every 8 connection(s)
I20260811 18:02:21.122457 28138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:21.178066 28138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38: Bootstrap starting.
I20260811 18:02:21.193014 28138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:21.197772 28138 log.cc:824] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38: Log is configured to *not* fsync() on all Append() calls
I20260811 18:02:21.214288 28138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38: No bootstrap required, opened a new log
I20260811 18:02:21.277101 28138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd094ee89344b97990298d22dd7ef38" member_type: VOTER }
I20260811 18:02:21.278034 28138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:21.278378 28138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd094ee89344b97990298d22dd7ef38, State: Initialized, Role: FOLLOWER
I20260811 18:02:21.280138 28138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd094ee89344b97990298d22dd7ef38" member_type: VOTER }
I20260811 18:02:21.281046 28138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:02:21.281373 28138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:02:21.281759 28138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:21.292096 28138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd094ee89344b97990298d22dd7ef38" member_type: VOTER }
I20260811 18:02:21.294178 28138 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd094ee89344b97990298d22dd7ef38; no voters:
I20260811 18:02:21.297158 28138 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:02:21.297904 28142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:02:21.303872 28142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd094ee89344b97990298d22dd7ef38, State: Running, Role: LEADER
I20260811 18:02:21.305472 28142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd094ee89344b97990298d22dd7ef38" member_type: VOTER }
I20260811 18:02:21.309585 28138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:02:21.332721 28147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bd094ee89344b97990298d22dd7ef38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd094ee89344b97990298d22dd7ef38" member_type: VOTER } }
I20260811 18:02:21.333990 28147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:21.337560 28146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd094ee89344b97990298d22dd7ef38. Latest consensus state: current_term: 1 leader_uuid: "6bd094ee89344b97990298d22dd7ef38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd094ee89344b97990298d22dd7ef38" member_type: VOTER } }
I20260811 18:02:21.340196 28146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:21.347087 28153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:02:21.383137 28153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:02:21.419281 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:02:21.458532 28153 catalog_manager.cc:1347] Generated new cluster ID: 87da592391f9476a8c3f03818a5ef9a1
I20260811 18:02:21.458979 28153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:02:21.501094 28153 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:02:21.513181 28153 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:02:21.542225 28153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38: Generated new TSK 0
I20260811 18:02:21.544257 28153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:02:21.579828 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:21.604861 28176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:21.614480 28177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:21.627580 28027 server_base.cc:1034] running on GCE node
W20260811 18:02:21.629647 28180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:21.631141 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:21.631385 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:21.631610 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471341631591 us; error 0 us; skew 500 ppm
I20260811 18:02:21.632715 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:21.643306 28027 webserver.cc:460] Webserver started at http://127.27.94.193:43059/ using document root <none> and password file <none>
I20260811 18:02:21.644722 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:21.645460 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:21.646201 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:21.648707 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "812d4c34acb84093b7c69ad194e1ae3d"
format_stamp: "Formatted at 2026-08-11 18:02:21 on dist-test-slave-kvfs"
I20260811 18:02:21.662067 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.008s
I20260811 18:02:21.670743 28191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:21.672118 28027 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s
I20260811 18:02:21.672482 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "812d4c34acb84093b7c69ad194e1ae3d"
format_stamp: "Formatted at 2026-08-11 18:02:21 on dist-test-slave-kvfs"
I20260811 18:02:21.672945 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:21.727701 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:21.731649 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:21.738899 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:21.750998 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:21.751399 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:21.751758 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:21.751941 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:22.146126 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:45617
I20260811 18:02:22.146327 28285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:45617 every 8 connection(s)
I20260811 18:02:22.181635 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:22.212994 28293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:22.225358 28296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:22.242664 28287 heartbeater.cc:347] Connected to a master server at 127.27.94.254:39985
I20260811 18:02:22.243774 28287 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:22.246719 28287 heartbeater.cc:511] Master 127.27.94.254:39985 requested a full tablet report, sending...
I20260811 18:02:22.254889 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:22.257764 28080 ts_manager.cc:194] Registered new tserver with Master: 812d4c34acb84093b7c69ad194e1ae3d (127.27.94.193:45617)
W20260811 18:02:22.261016 28303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:22.262943 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:22.263576 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:22.263870 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471342263835 us; error 0 us; skew 500 ppm
I20260811 18:02:22.265463 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:22.265729 28080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51734
I20260811 18:02:22.274078 28027 webserver.cc:460] Webserver started at http://127.27.94.194:42055/ using document root <none> and password file <none>
I20260811 18:02:22.275202 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:22.275596 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:22.276274 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:22.278663 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/ts-1-root/instance:
uuid: "a3d83e4d3a634176aadd4d19538f1719"
format_stamp: "Formatted at 2026-08-11 18:02:22 on dist-test-slave-kvfs"
I20260811 18:02:22.288271 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20260811 18:02:22.296562 28307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:22.298580 28027 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260811 18:02:22.299108 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/ts-1-root
uuid: "a3d83e4d3a634176aadd4d19538f1719"
format_stamp: "Formatted at 2026-08-11 18:02:22 on dist-test-slave-kvfs"
I20260811 18:02:22.299527 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1786471338663725-28027-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:22.363207 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:22.366608 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:22.373519 28287 heartbeater.cc:503] Master 127.27.94.254:39985 was elected leader, sending a full tablet report...
I20260811 18:02:22.377969 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:22.389604 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:22.389956 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:22.390241 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:22.390426 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:22.806404 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.194:46317
I20260811 18:02:22.807075 28402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.194:46317 every 8 connection(s)
I20260811 18:02:22.861703 28403 heartbeater.cc:347] Connected to a master server at 127.27.94.254:39985
I20260811 18:02:22.862342 28403 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:22.863935 28403 heartbeater.cc:511] Master 127.27.94.254:39985 requested a full tablet report, sending...
I20260811 18:02:22.867451 28080 ts_manager.cc:194] Registered new tserver with Master: a3d83e4d3a634176aadd4d19538f1719 (127.27.94.194:46317)
I20260811 18:02:22.867894 28027 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.044721196s
I20260811 18:02:22.869829 28080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51744
I20260811 18:02:22.978681 28403 heartbeater.cc:503] Master 127.27.94.254:39985 was elected leader, sending a full tablet report...
I20260811 18:02:22.951622 28080 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20260811 18:02:23.002094 28080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260811 18:02:23.167892 28234 tablet_service.cc:1467] Processing CreateTablet for tablet 6d048499e70e4b2fa9cb440a4843278b (DEFAULT_TABLE table=test-table [id=b4b610b588814458abbc526021997ee2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:23.168422 28232 tablet_service.cc:1467] Processing CreateTablet for tablet 36aa2f9682a94b99a3b464956cd201ae (DEFAULT_TABLE table=test-table [id=b4b610b588814458abbc526021997ee2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:23.170123 28234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d048499e70e4b2fa9cb440a4843278b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:23.172278 28232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36aa2f9682a94b99a3b464956cd201ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:23.176523 28353 tablet_service.cc:1467] Processing CreateTablet for tablet 6d048499e70e4b2fa9cb440a4843278b (DEFAULT_TABLE table=test-table [id=b4b610b588814458abbc526021997ee2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:23.176939 28352 tablet_service.cc:1467] Processing CreateTablet for tablet 36aa2f9682a94b99a3b464956cd201ae (DEFAULT_TABLE table=test-table [id=b4b610b588814458abbc526021997ee2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:23.180608 28353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d048499e70e4b2fa9cb440a4843278b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:23.182053 28352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36aa2f9682a94b99a3b464956cd201ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:23.228783 28432 tablet_bootstrap.cc:492] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719: Bootstrap starting.
I20260811 18:02:23.271587 28433 tablet_bootstrap.cc:492] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d: Bootstrap starting.
I20260811 18:02:23.272256 28432 tablet_bootstrap.cc:654] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:23.291831 28433 tablet_bootstrap.cc:654] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:23.300364 28432 tablet_bootstrap.cc:492] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719: No bootstrap required, opened a new log
I20260811 18:02:23.301648 28432 ts_tablet_manager.cc:1397] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719: Time spent bootstrapping tablet: real 0.077s user 0.022s sys 0.015s
I20260811 18:02:23.312690 28432 raft_consensus.cc:348] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:23.314155 28432 raft_consensus.cc:729] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d83e4d3a634176aadd4d19538f1719, State: Initialized, Role: FOLLOWER
I20260811 18:02:23.314983 28432 consensus_queue.cc:260] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:23.315745 28433 tablet_bootstrap.cc:492] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d: No bootstrap required, opened a new log
I20260811 18:02:23.316529 28433 ts_tablet_manager.cc:1397] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.027s
I20260811 18:02:23.322616 28433 raft_consensus.cc:348] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:23.323484 28433 raft_consensus.cc:729] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812d4c34acb84093b7c69ad194e1ae3d, State: Initialized, Role: FOLLOWER
I20260811 18:02:23.324604 28432 ts_tablet_manager.cc:1428] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719: Time spent starting tablet: real 0.022s user 0.010s sys 0.008s
I20260811 18:02:23.324288 28433 consensus_queue.cc:260] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
W20260811 18:02:23.325560 28404 tablet.cc:2369] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 18:02:23.326068 28432 tablet_bootstrap.cc:492] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719: Bootstrap starting.
I20260811 18:02:23.333262 28433 ts_tablet_manager.cc:1428] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260811 18:02:23.335597 28433 tablet_bootstrap.cc:492] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d: Bootstrap starting.
I20260811 18:02:23.349586 28433 tablet_bootstrap.cc:654] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:23.366236 28432 tablet_bootstrap.cc:654] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:23.378841 28433 tablet_bootstrap.cc:492] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d: No bootstrap required, opened a new log
I20260811 18:02:23.379729 28433 ts_tablet_manager.cc:1397] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.037s
I20260811 18:02:23.384826 28433 raft_consensus.cc:348] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:23.386008 28433 raft_consensus.cc:729] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812d4c34acb84093b7c69ad194e1ae3d, State: Initialized, Role: FOLLOWER
I20260811 18:02:23.386909 28433 consensus_queue.cc:260] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:23.390017 28433 ts_tablet_manager.cc:1428] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260811 18:02:23.396325 28432 tablet_bootstrap.cc:492] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719: No bootstrap required, opened a new log
I20260811 18:02:23.396773 28432 ts_tablet_manager.cc:1397] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719: Time spent bootstrapping tablet: real 0.071s user 0.017s sys 0.024s
I20260811 18:02:23.400841 28432 raft_consensus.cc:348] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:23.401824 28432 raft_consensus.cc:729] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d83e4d3a634176aadd4d19538f1719, State: Initialized, Role: FOLLOWER
I20260811 18:02:23.402675 28432 consensus_queue.cc:260] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:23.406070 28432 ts_tablet_manager.cc:1428] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20260811 18:02:23.443148 28027 ts_tablet_manager-itest.cc:367] Electing peer 2...
I20260811 18:02:23.443542 28027 raft_consensus.cc:412] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 0 FOLLOWER]: Emulating election...
I20260811 18:02:23.443845 28027 raft_consensus.cc:3037] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:23.450240 28027 raft_consensus.cc:686] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 1 LEADER]: Becoming Leader. State: Replica: a3d83e4d3a634176aadd4d19538f1719, State: Running, Role: LEADER
I20260811 18:02:23.450985 28027 consensus_queue.cc:237] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:23.463793 28027 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260811 18:02:23.470090 28077 catalog_manager.cc:5495] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3d83e4d3a634176aadd4d19538f1719 (127.27.94.194). New cstate: current_term: 1 leader_uuid: "a3d83e4d3a634176aadd4d19538f1719" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260811 18:02:23.514325 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260811 18:02:23.620508 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260811 18:02:23.826421 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260811 18:02:23.944700 28250 raft_consensus.cc:3037] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:23.954592 28436 consensus_queue.cc:1035] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [LEADER]: Connected to new peer: Peer: permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 18:02:24.132366 28027 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260811 18:02:24.132678 28027 raft_consensus.cc:412] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 0 FOLLOWER]: Emulating election...
I20260811 18:02:24.132918 28027 raft_consensus.cc:3037] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:24.137586 28027 raft_consensus.cc:686] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 1 LEADER]: Becoming Leader. State: Replica: a3d83e4d3a634176aadd4d19538f1719, State: Running, Role: LEADER
I20260811 18:02:24.138383 28027 consensus_queue.cc:237] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:24.140985 28027 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260811 18:02:24.147194 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260811 18:02:24.148761 28080 catalog_manager.cc:5495] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3d83e4d3a634176aadd4d19538f1719 (127.27.94.194). New cstate: current_term: 1 leader_uuid: "a3d83e4d3a634176aadd4d19538f1719" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260811 18:02:24.253512 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260811 18:02:24.458846 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260811 18:02:24.546769 28250 raft_consensus.cc:3037] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:24.560833 28454 consensus_queue.cc:1035] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [LEADER]: Connected to new peer: Peer: permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 18:02:24.965464 28027 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260811 18:02:24.965745 28027 raft_consensus.cc:412] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 1 LEADER]: Emulating election...
I20260811 18:02:24.966027 28027 raft_consensus.cc:3032] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 1 LEADER]: Stepping down as leader of term 1
I20260811 18:02:24.966679 28027 raft_consensus.cc:729] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3d83e4d3a634176aadd4d19538f1719, State: Running, Role: LEADER
I20260811 18:02:24.967237 28027 consensus_queue.cc:260] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:24.968070 28027 raft_consensus.cc:3037] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:24.972075 28027 raft_consensus.cc:686] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 2 LEADER]: Becoming Leader. State: Replica: a3d83e4d3a634176aadd4d19538f1719, State: Running, Role: LEADER
I20260811 18:02:24.972730 28027 consensus_queue.cc:237] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:24.974350 28027 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260811 18:02:24.978865 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260811 18:02:24.978628 28080 catalog_manager.cc:5495] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a3d83e4d3a634176aadd4d19538f1719" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260811 18:02:25.083534 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260811 18:02:25.289179 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260811 18:02:25.547565 28250 raft_consensus.cc:3037] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:25.555174 28250 raft_consensus.cc:1261] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 2 FOLLOWER]: Refusing update from remote peer a3d83e4d3a634176aadd4d19538f1719: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260811 18:02:25.557008 28454 consensus_queue.cc:1035] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [LEADER]: Connected to new peer: Peer: permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260811 18:02:25.595803 28027 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260811 18:02:25.596084 28027 raft_consensus.cc:412] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 2 FOLLOWER]: Emulating election...
I20260811 18:02:25.596277 28027 raft_consensus.cc:3037] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:25.601311 28027 raft_consensus.cc:686] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 3 LEADER]: Becoming Leader. State: Replica: 812d4c34acb84093b7c69ad194e1ae3d, State: Running, Role: LEADER
I20260811 18:02:25.602192 28027 consensus_queue.cc:237] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:25.604755 28027 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260811 18:02:25.611603 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260811 18:02:25.612840 28077 catalog_manager.cc:5495] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d reported cstate change: term changed from 2 to 3, leader changed from a3d83e4d3a634176aadd4d19538f1719 (127.27.94.194) to 812d4c34acb84093b7c69ad194e1ae3d (127.27.94.193). New cstate: current_term: 3 leader_uuid: "812d4c34acb84093b7c69ad194e1ae3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260811 18:02:25.718466 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260811 18:02:25.924005 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260811 18:02:26.129408 28366 raft_consensus.cc:3032] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 2 LEADER]: Stepping down as leader of term 2
I20260811 18:02:26.130836 28366 raft_consensus.cc:729] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a3d83e4d3a634176aadd4d19538f1719, State: Running, Role: LEADER
I20260811 18:02:26.131533 28366 consensus_queue.cc:260] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:26.132463 28366 raft_consensus.cc:3037] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:26.138233 28366 raft_consensus.cc:1261] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 3 FOLLOWER]: Refusing update from remote peer 812d4c34acb84093b7c69ad194e1ae3d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260811 18:02:26.139638 28483 consensus_queue.cc:1035] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260811 18:02:26.430159 28027 ts_tablet_manager-itest.cc:367] Electing peer 1...
I20260811 18:02:26.430495 28027 raft_consensus.cc:412] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [term 1 FOLLOWER]: Emulating election...
I20260811 18:02:26.431288 28027 raft_consensus.cc:3037] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:26.437680 28027 raft_consensus.cc:686] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [term 2 LEADER]: Becoming Leader. State: Replica: 812d4c34acb84093b7c69ad194e1ae3d, State: Running, Role: LEADER
I20260811 18:02:26.438473 28027 consensus_queue.cc:237] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:26.440665 28027 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260811 18:02:26.445724 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260811 18:02:26.444888 28080 catalog_manager.cc:5495] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d reported cstate change: term changed from 1 to 2, leader changed from a3d83e4d3a634176aadd4d19538f1719 (127.27.94.194) to 812d4c34acb84093b7c69ad194e1ae3d (127.27.94.193). New cstate: current_term: 2 leader_uuid: "812d4c34acb84093b7c69ad194e1ae3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } health_report { overall_health: UNKNOWN } } }
I20260811 18:02:26.553905 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260811 18:02:26.758687 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260811 18:02:26.831476 28250 raft_consensus.cc:1226] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [term 2 LEADER]: Rejecting Update request from peer a3d83e4d3a634176aadd4d19538f1719 for earlier term 1. Current term is 2. Ops: []
I20260811 18:02:26.833493 28501 consensus_queue.cc:1046] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260811 18:02:26.836906 28501 consensus_queue.cc:1225] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [LEADER]: Peer 812d4c34acb84093b7c69ad194e1ae3d log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260811 18:02:26.837595 28502 raft_consensus.cc:3032] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 1 LEADER]: Stepping down as leader of term 1
I20260811 18:02:26.837922 28502 raft_consensus.cc:729] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3d83e4d3a634176aadd4d19538f1719, State: Running, Role: LEADER
I20260811 18:02:26.838802 28502 consensus_queue.cc:260] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:26.839996 28502 raft_consensus.cc:3037] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:26.878989 28366 raft_consensus.cc:1261] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 2 FOLLOWER]: Refusing update from remote peer 812d4c34acb84093b7c69ad194e1ae3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260811 18:02:26.880714 28486 consensus_queue.cc:1035] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260811 18:02:27.065201 28027 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260811 18:02:27.065526 28027 raft_consensus.cc:412] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 3 FOLLOWER]: Emulating election...
I20260811 18:02:27.065832 28027 raft_consensus.cc:3037] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:02:27.070921 28027 raft_consensus.cc:686] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 4 LEADER]: Becoming Leader. State: Replica: a3d83e4d3a634176aadd4d19538f1719, State: Running, Role: LEADER
I20260811 18:02:27.071575 28027 consensus_queue.cc:237] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:27.073086 28027 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260811 18:02:27.077535 28080 catalog_manager.cc:5495] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 reported cstate change: term changed from 3 to 4, leader changed from 812d4c34acb84093b7c69ad194e1ae3d (127.27.94.193) to a3d83e4d3a634176aadd4d19538f1719 (127.27.94.194). New cstate: current_term: 4 leader_uuid: "a3d83e4d3a634176aadd4d19538f1719" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } health_report { overall_health: HEALTHY } } }
I20260811 18:02:27.078790 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260811 18:02:27.143095 28366 raft_consensus.cc:1226] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 4 LEADER]: Rejecting Update request from peer 812d4c34acb84093b7c69ad194e1ae3d for earlier term 3. Current term is 4. Ops: []
I20260811 18:02:27.145543 28505 consensus_queue.cc:1046] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260811 18:02:27.146409 28505 consensus_queue.cc:1225] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [LEADER]: Peer a3d83e4d3a634176aadd4d19538f1719 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260811 18:02:27.146885 28486 raft_consensus.cc:3032] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 3 LEADER]: Stepping down as leader of term 3
I20260811 18:02:27.147218 28486 raft_consensus.cc:729] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 812d4c34acb84093b7c69ad194e1ae3d, State: Running, Role: LEADER
I20260811 18:02:27.148492 28486 consensus_queue.cc:260] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 } } peers { permanent_uuid: "a3d83e4d3a634176aadd4d19538f1719" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 46317 } }
I20260811 18:02:27.149780 28486 raft_consensus.cc:3037] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:02:27.185187 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260811 18:02:27.391940 28027 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260811 18:02:27.637359 28250 raft_consensus.cc:1261] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 4 FOLLOWER]: Refusing update from remote peer a3d83e4d3a634176aadd4d19538f1719: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260811 18:02:27.638903 28519 consensus_queue.cc:1035] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [LEADER]: Connected to new peer: Peer: permanent_uuid: "812d4c34acb84093b7c69ad194e1ae3d" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260811 18:02:27.901530 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:02:27.929623 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:27.930403 28027 tablet_replica.cc:330] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d: stopping tablet replica
I20260811 18:02:27.931166 28027 raft_consensus.cc:2229] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260811 18:02:27.931680 28027 raft_consensus.cc:2258] T 6d048499e70e4b2fa9cb440a4843278b P 812d4c34acb84093b7c69ad194e1ae3d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:27.934899 28027 tablet_replica.cc:330] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d: stopping tablet replica
I20260811 18:02:27.935366 28027 raft_consensus.cc:2229] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [term 2 LEADER]: Raft consensus shutting down.
I20260811 18:02:27.936115 28027 raft_consensus.cc:2258] T 36aa2f9682a94b99a3b464956cd201ae P 812d4c34acb84093b7c69ad194e1ae3d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:28.167883 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:02:28.178049 28027 tablet_server.cc:178] TabletServer@127.27.94.194:0 shutting down...
I20260811 18:02:28.208572 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:28.210628 28027 tablet_replica.cc:330] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719: stopping tablet replica
I20260811 18:02:28.211306 28027 raft_consensus.cc:2229] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260811 18:02:28.211843 28027 raft_consensus.cc:2258] T 36aa2f9682a94b99a3b464956cd201ae P a3d83e4d3a634176aadd4d19538f1719 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:28.213316 28027 tablet_replica.cc:330] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719: stopping tablet replica
I20260811 18:02:28.213769 28027 raft_consensus.cc:2229] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 4 LEADER]: Raft consensus shutting down.
I20260811 18:02:28.214354 28027 raft_consensus.cc:2258] T 6d048499e70e4b2fa9cb440a4843278b P a3d83e4d3a634176aadd4d19538f1719 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:28.235589 28027 tablet_server.cc:194] TabletServer@127.27.94.194:0 shutdown complete.
I20260811 18:02:28.245472 28027 master.cc:537] Master@127.27.94.254:39985 shutting down...
I20260811 18:02:28.503276 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:02:28.503944 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:28.504611 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6bd094ee89344b97990298d22dd7ef38: stopping tablet replica
I20260811 18:02:28.523332 28027 master.cc:552] Master@127.27.94.254:39985 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (9581 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20260811 18:02:28.557389 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:34743
I20260811 18:02:28.558485 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:28.564445 28534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:28.566490 28536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:28.567894 28540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:28.568100 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:28.568867 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:28.569038 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:28.569209 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471348569195 us; error 0 us; skew 500 ppm
I20260811 18:02:28.569788 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:28.572667 28027 webserver.cc:460] Webserver started at http://127.27.94.254:37193/ using document root <none> and password file <none>
I20260811 18:02:28.573618 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:28.574079 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:28.574674 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:28.576143 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "3fd4db3da5bf466db92f0d1c370fd0f4"
format_stamp: "Formatted at 2026-08-11 18:02:28 on dist-test-slave-kvfs"
I20260811 18:02:28.581271 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 18:02:28.585312 28546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:28.586211 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 18:02:28.586534 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "3fd4db3da5bf466db92f0d1c370fd0f4"
format_stamp: "Formatted at 2026-08-11 18:02:28 on dist-test-slave-kvfs"
I20260811 18:02:28.586827 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:28.603444 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:28.604931 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:28.656461 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:34743
I20260811 18:02:28.656579 28626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:34743 every 8 connection(s)
I20260811 18:02:28.660544 28628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:28.673648 28628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4: Bootstrap starting.
I20260811 18:02:28.679804 28628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:28.685390 28628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4: No bootstrap required, opened a new log
I20260811 18:02:28.687486 28628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd4db3da5bf466db92f0d1c370fd0f4" member_type: VOTER }
I20260811 18:02:28.687985 28628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:28.688407 28628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd4db3da5bf466db92f0d1c370fd0f4, State: Initialized, Role: FOLLOWER
I20260811 18:02:28.688990 28628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd4db3da5bf466db92f0d1c370fd0f4" member_type: VOTER }
I20260811 18:02:28.689556 28628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:02:28.689770 28628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:02:28.689996 28628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:28.695932 28628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd4db3da5bf466db92f0d1c370fd0f4" member_type: VOTER }
I20260811 18:02:28.696782 28628 leader_election.cc:302] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fd4db3da5bf466db92f0d1c370fd0f4; no voters:
I20260811 18:02:28.698442 28628 leader_election.cc:288] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:02:28.698915 28633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:02:28.700660 28633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3fd4db3da5bf466db92f0d1c370fd0f4, State: Running, Role: LEADER
I20260811 18:02:28.701460 28633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd4db3da5bf466db92f0d1c370fd0f4" member_type: VOTER }
I20260811 18:02:28.702760 28628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:02:28.704576 28634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fd4db3da5bf466db92f0d1c370fd0f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd4db3da5bf466db92f0d1c370fd0f4" member_type: VOTER } }
I20260811 18:02:28.704643 28635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fd4db3da5bf466db92f0d1c370fd0f4. Latest consensus state: current_term: 1 leader_uuid: "3fd4db3da5bf466db92f0d1c370fd0f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd4db3da5bf466db92f0d1c370fd0f4" member_type: VOTER } }
I20260811 18:02:28.705395 28634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:28.705461 28635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:28.710485 28640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:02:28.715987 28640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:02:28.726684 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:02:28.730376 28640 catalog_manager.cc:1347] Generated new cluster ID: a4b8dc0a4972435287dd214a492e0f5b
I20260811 18:02:28.730748 28640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:02:28.757730 28640 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:02:28.759912 28640 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:02:28.787281 28640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4: Generated new TSK 0
I20260811 18:02:28.788110 28640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:02:28.796955 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:28.804103 28660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:28.804880 28662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:28.809906 28671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:28.814896 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:28.815860 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:28.816133 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:28.816257 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471348816248 us; error 0 us; skew 500 ppm
I20260811 18:02:28.816831 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:28.821311 28027 webserver.cc:460] Webserver started at http://127.27.94.193:36049/ using document root <none> and password file <none>
I20260811 18:02:28.821841 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:28.822003 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:28.822338 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:28.823917 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "16c9ce660a444841950203661e6436d0"
format_stamp: "Formatted at 2026-08-11 18:02:28 on dist-test-slave-kvfs"
I20260811 18:02:28.829067 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260811 18:02:28.833034 28685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:28.834156 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 18:02:28.834561 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "16c9ce660a444841950203661e6436d0"
format_stamp: "Formatted at 2026-08-11 18:02:28 on dist-test-slave-kvfs"
I20260811 18:02:28.834852 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:28.851608 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:28.853339 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:28.854888 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:28.857877 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:28.858158 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:28.858445 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:28.858664 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:28.919548 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:43945
I20260811 18:02:28.919642 28794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:43945 every 8 connection(s)
I20260811 18:02:28.925583 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:28.939590 28808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:28.941362 28809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:28.945710 28812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:28.947603 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:28.949410 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260811 18:02:28.949616 28796 heartbeater.cc:347] Connected to a master server at 127.27.94.254:34743
W20260811 18:02:28.949766 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:28.950358 28796 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:28.950222 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471348950130 us; error 0 us; skew 500 ppm
I20260811 18:02:28.951206 28796 heartbeater.cc:511] Master 127.27.94.254:34743 requested a full tablet report, sending...
I20260811 18:02:28.951252 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:28.953727 28574 ts_manager.cc:194] Registered new tserver with Master: 16c9ce660a444841950203661e6436d0 (127.27.94.193:43945)
I20260811 18:02:28.954994 28027 webserver.cc:460] Webserver started at http://127.27.94.194:40065/ using document root <none> and password file <none>
I20260811 18:02:28.955801 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:28.956115 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:28.956482 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:28.957031 28574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60294
I20260811 18:02:28.958221 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/ts-1-root/instance:
uuid: "91cf684e0cb74671a0148d524a2ee99a"
format_stamp: "Formatted at 2026-08-11 18:02:28 on dist-test-slave-kvfs"
I20260811 18:02:28.963112 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260811 18:02:28.967664 28819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:28.968797 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 18:02:28.969129 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/ts-1-root
uuid: "91cf684e0cb74671a0148d524a2ee99a"
format_stamp: "Formatted at 2026-08-11 18:02:28 on dist-test-slave-kvfs"
I20260811 18:02:28.969455 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1786471338663725-28027-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:29.034987 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:29.037424 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:29.039264 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:29.042239 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:29.042503 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:29.042826 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:29.043006 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:29.061182 28796 heartbeater.cc:503] Master 127.27.94.254:34743 was elected leader, sending a full tablet report...
I20260811 18:02:29.101992 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.194:46031
I20260811 18:02:29.102279 28929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.194:46031 every 8 connection(s)
I20260811 18:02:29.120733 28931 heartbeater.cc:347] Connected to a master server at 127.27.94.254:34743
I20260811 18:02:29.121356 28931 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:29.122350 28931 heartbeater.cc:511] Master 127.27.94.254:34743 requested a full tablet report, sending...
I20260811 18:02:29.124454 28574 ts_manager.cc:194] Registered new tserver with Master: 91cf684e0cb74671a0148d524a2ee99a (127.27.94.194:46031)
I20260811 18:02:29.124755 28027 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.018648327s
I20260811 18:02:29.126335 28574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60302
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:421: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20260811 18:02:29.158730 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:02:29.183303 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:29.200910 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:02:29.211386 28027 tablet_server.cc:178] TabletServer@127.27.94.194:0 shutting down...
I20260811 18:02:29.229139 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:29.246668 28027 tablet_server.cc:194] TabletServer@127.27.94.194:0 shutdown complete.
I20260811 18:02:29.257112 28027 master.cc:537] Master@127.27.94.254:34743 shutting down...
I20260811 18:02:29.285778 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:02:29.286530 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:29.287014 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3fd4db3da5bf466db92f0d1c370fd0f4: stopping tablet replica
I20260811 18:02:29.307344 28027 master.cc:552] Master@127.27.94.254:34743 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (782 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20260811 18:02:29.338752 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:39917
I20260811 18:02:29.339998 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:29.347770 28959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:29.348534 28961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:29.349913 28967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:29.352206 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:29.352924 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:29.353302 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:29.353509 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471349353494 us; error 0 us; skew 500 ppm
I20260811 18:02:29.354074 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:29.357075 28027 webserver.cc:460] Webserver started at http://127.27.94.254:40393/ using document root <none> and password file <none>
I20260811 18:02:29.357535 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:29.357682 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:29.357923 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:29.359146 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "cbb5487d349f41c0a823832332410ba3"
format_stamp: "Formatted at 2026-08-11 18:02:29 on dist-test-slave-kvfs"
I20260811 18:02:29.365010 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260811 18:02:29.369627 28976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:29.370901 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 18:02:29.371347 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "cbb5487d349f41c0a823832332410ba3"
format_stamp: "Formatted at 2026-08-11 18:02:29 on dist-test-slave-kvfs"
I20260811 18:02:29.371990 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:29.396576 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:29.397904 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:29.443989 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:39917
I20260811 18:02:29.444168 29064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:39917 every 8 connection(s)
I20260811 18:02:29.448278 29067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:29.460364 29067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3: Bootstrap starting.
I20260811 18:02:29.466171 29067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:29.471660 29067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3: No bootstrap required, opened a new log
I20260811 18:02:29.474295 29067 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb5487d349f41c0a823832332410ba3" member_type: VOTER }
I20260811 18:02:29.474977 29067 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:29.475167 29067 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb5487d349f41c0a823832332410ba3, State: Initialized, Role: FOLLOWER
I20260811 18:02:29.475719 29067 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb5487d349f41c0a823832332410ba3" member_type: VOTER }
I20260811 18:02:29.476516 29067 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:02:29.476809 29067 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:02:29.477109 29067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:29.483635 29067 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb5487d349f41c0a823832332410ba3" member_type: VOTER }
I20260811 18:02:29.484489 29067 leader_election.cc:302] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbb5487d349f41c0a823832332410ba3; no voters:
I20260811 18:02:29.486163 29067 leader_election.cc:288] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:02:29.486797 29071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:02:29.489619 29067 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:02:29.489337 29071 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 1 LEADER]: Becoming Leader. State: Replica: cbb5487d349f41c0a823832332410ba3, State: Running, Role: LEADER
I20260811 18:02:29.490556 29071 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb5487d349f41c0a823832332410ba3" member_type: VOTER }
I20260811 18:02:29.496850 29074 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbb5487d349f41c0a823832332410ba3. Latest consensus state: current_term: 1 leader_uuid: "cbb5487d349f41c0a823832332410ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb5487d349f41c0a823832332410ba3" member_type: VOTER } }
I20260811 18:02:29.497507 29074 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:29.496448 29073 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbb5487d349f41c0a823832332410ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb5487d349f41c0a823832332410ba3" member_type: VOTER } }
I20260811 18:02:29.497994 29073 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:29.501655 29087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:02:29.511940 29087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:02:29.513410 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:02:29.522351 29087 catalog_manager.cc:1347] Generated new cluster ID: d1bda2828ec843ca847e4a5ab9ea09cf
I20260811 18:02:29.522878 29087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:02:29.544099 29087 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:02:29.545961 29087 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:02:29.560339 29087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3: Generated new TSK 0
I20260811 18:02:29.561081 29087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:02:29.580446 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:29.588965 29105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:29.591013 29106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:29.592876 29108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:29.593115 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:29.594225 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:29.594830 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:29.595045 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471349595031 us; error 0 us; skew 500 ppm
I20260811 18:02:29.595597 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:29.598835 28027 webserver.cc:460] Webserver started at http://127.27.94.193:33477/ using document root <none> and password file <none>
I20260811 18:02:29.599313 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:29.599470 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:29.599682 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:29.601037 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "0c80a2721b0141958cdb190fcfe04e1c"
format_stamp: "Formatted at 2026-08-11 18:02:29 on dist-test-slave-kvfs"
I20260811 18:02:29.606446 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260811 18:02:29.611562 29116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:29.612748 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 18:02:29.613157 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "0c80a2721b0141958cdb190fcfe04e1c"
format_stamp: "Formatted at 2026-08-11 18:02:29 on dist-test-slave-kvfs"
I20260811 18:02:29.613483 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:29.633760 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:29.635120 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:29.636967 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:29.639804 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:29.640167 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260811 18:02:29.640437 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:29.640604 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:29.700373 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:44143
I20260811 18:02:29.700477 29219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:44143 every 8 connection(s)
I20260811 18:02:29.705622 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:29.715214 29229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:29.716398 29230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:29.720417 28027 server_base.cc:1034] running on GCE node
W20260811 18:02:29.720928 29236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:29.722800 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:29.723033 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:29.723083 29220 heartbeater.cc:347] Connected to a master server at 127.27.94.254:39917
I20260811 18:02:29.723302 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471349723258 us; error 0 us; skew 500 ppm
I20260811 18:02:29.723472 29220 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:29.724071 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:29.724356 29220 heartbeater.cc:511] Master 127.27.94.254:39917 requested a full tablet report, sending...
I20260811 18:02:29.727229 28027 webserver.cc:460] Webserver started at http://127.27.94.194:42837/ using document root <none> and password file <none>
I20260811 18:02:29.727269 29001 ts_manager.cc:194] Registered new tserver with Master: 0c80a2721b0141958cdb190fcfe04e1c (127.27.94.193:44143)
I20260811 18:02:29.728014 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:29.728336 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:29.728659 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:29.730230 29001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42022
I20260811 18:02:29.730353 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/ts-1-root/instance:
uuid: "e57b7b81ddf3403cbd1931ec1e63caee"
format_stamp: "Formatted at 2026-08-11 18:02:29 on dist-test-slave-kvfs"
I20260811 18:02:29.738867 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.010s sys 0.000s
I20260811 18:02:29.743410 29245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:29.744541 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 18:02:29.744985 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/ts-1-root
uuid: "e57b7b81ddf3403cbd1931ec1e63caee"
format_stamp: "Formatted at 2026-08-11 18:02:29 on dist-test-slave-kvfs"
I20260811 18:02:29.745298 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1786471338663725-28027-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:29.758241 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:29.759521 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:29.761184 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:29.763552 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:29.763796 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:29.764171 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:29.764441 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:29.813745 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.194:33133
I20260811 18:02:29.813808 29344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.194:33133 every 8 connection(s)
I20260811 18:02:29.831337 29345 heartbeater.cc:347] Connected to a master server at 127.27.94.254:39917
I20260811 18:02:29.831725 29345 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:29.832811 29345 heartbeater.cc:511] Master 127.27.94.254:39917 requested a full tablet report, sending...
I20260811 18:02:29.834862 29001 ts_manager.cc:194] Registered new tserver with Master: e57b7b81ddf3403cbd1931ec1e63caee (127.27.94.194:33133)
I20260811 18:02:29.835265 29220 heartbeater.cc:503] Master 127.27.94.254:39917 was elected leader, sending a full tablet report...
I20260811 18:02:29.835939 28027 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.01868804s
I20260811 18:02:29.836843 29001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42028
I20260811 18:02:29.866338 28999 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 18:02:29.868276 28999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260811 18:02:29.912609 29170 tablet_service.cc:1467] Processing CreateTablet for tablet c7edabe8118e4511a06409a1b420462e (DEFAULT_TABLE table=test-table [id=6e07cf4a15cd448696cc2a2c6624d7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:29.914222 29170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7edabe8118e4511a06409a1b420462e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:29.914012 29286 tablet_service.cc:1467] Processing CreateTablet for tablet c7edabe8118e4511a06409a1b420462e (DEFAULT_TABLE table=test-table [id=6e07cf4a15cd448696cc2a2c6624d7e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:29.915369 29286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7edabe8118e4511a06409a1b420462e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:29.932569 29367 tablet_bootstrap.cc:492] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c: Bootstrap starting.
I20260811 18:02:29.936316 29368 tablet_bootstrap.cc:492] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee: Bootstrap starting.
I20260811 18:02:29.940418 29345 heartbeater.cc:503] Master 127.27.94.254:39917 was elected leader, sending a full tablet report...
I20260811 18:02:29.940694 29367 tablet_bootstrap.cc:654] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:29.945039 29368 tablet_bootstrap.cc:654] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:29.954535 29367 tablet_bootstrap.cc:492] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c: No bootstrap required, opened a new log
I20260811 18:02:29.955035 29367 ts_tablet_manager.cc:1397] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.014s
I20260811 18:02:29.956430 29368 tablet_bootstrap.cc:492] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee: No bootstrap required, opened a new log
I20260811 18:02:29.956894 29368 ts_tablet_manager.cc:1397] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20260811 18:02:29.957505 29367 raft_consensus.cc:348] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 } } peers { permanent_uuid: "0c80a2721b0141958cdb190fcfe04e1c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 44143 } }
I20260811 18:02:29.958081 29367 raft_consensus.cc:729] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c80a2721b0141958cdb190fcfe04e1c, State: Initialized, Role: FOLLOWER
I20260811 18:02:29.958565 29367 consensus_queue.cc:260] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 } } peers { permanent_uuid: "0c80a2721b0141958cdb190fcfe04e1c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 44143 } }
I20260811 18:02:29.959270 29368 raft_consensus.cc:348] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 } } peers { permanent_uuid: "0c80a2721b0141958cdb190fcfe04e1c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 44143 } }
I20260811 18:02:29.959860 29368 raft_consensus.cc:729] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57b7b81ddf3403cbd1931ec1e63caee, State: Initialized, Role: FOLLOWER
I20260811 18:02:29.960453 29368 consensus_queue.cc:260] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 } } peers { permanent_uuid: "0c80a2721b0141958cdb190fcfe04e1c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 44143 } }
I20260811 18:02:29.962626 29367 ts_tablet_manager.cc:1428] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 18:02:29.964308 29368 ts_tablet_manager.cc:1428] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260811 18:02:30.003518 28027 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260811 18:02:30.003805 28027 raft_consensus.cc:412] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 0 FOLLOWER]: Emulating election...
I20260811 18:02:30.004124 28027 raft_consensus.cc:3037] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:30.010568 28027 raft_consensus.cc:686] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 1 LEADER]: Becoming Leader. State: Replica: 0c80a2721b0141958cdb190fcfe04e1c, State: Running, Role: LEADER
W20260811 18:02:30.011040 29220 heartbeater.cc:644] Failed to heartbeat to 127.27.94.254:39917 (0 consecutive failures): IO error: failing all heartbeats for tests
I20260811 18:02:30.011641 28027 consensus_queue.cc:237] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 } } peers { permanent_uuid: "0c80a2721b0141958cdb190fcfe04e1c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 44143 } }
I20260811 18:02:30.015113 28027 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260811 18:02:30.041261 28027 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260811 18:02:30.146310 28027 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260811 18:02:30.352886 28027 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260811 18:02:30.423794 29300 raft_consensus.cc:3037] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:30.430552 29372 consensus_queue.cc:1035] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 18:02:30.658495 28027 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260811 18:02:30.658732 28027 raft_consensus.cc:412] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 1 LEADER]: Emulating election...
I20260811 18:02:30.658985 28027 raft_consensus.cc:3032] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 1 LEADER]: Stepping down as leader of term 1
I20260811 18:02:30.659355 28027 raft_consensus.cc:729] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c80a2721b0141958cdb190fcfe04e1c, State: Running, Role: LEADER
I20260811 18:02:30.659909 28027 consensus_queue.cc:260] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 } } peers { permanent_uuid: "0c80a2721b0141958cdb190fcfe04e1c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 44143 } }
I20260811 18:02:30.660701 28027 raft_consensus.cc:3037] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:30.665277 28027 raft_consensus.cc:686] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 2 LEADER]: Becoming Leader. State: Replica: 0c80a2721b0141958cdb190fcfe04e1c, State: Running, Role: LEADER
I20260811 18:02:30.665853 28027 consensus_queue.cc:237] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 } } peers { permanent_uuid: "0c80a2721b0141958cdb190fcfe04e1c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 44143 } }
I20260811 18:02:30.667230 28027 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260811 18:02:30.671128 28027 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260811 18:02:30.776289 28027 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260811 18:02:30.980665 28027 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260811 18:02:31.089233 29300 raft_consensus.cc:3037] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:31.095409 29300 raft_consensus.cc:1261] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee [term 2 FOLLOWER]: Refusing update from remote peer 0c80a2721b0141958cdb190fcfe04e1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260811 18:02:31.097018 29372 consensus_queue.cc:1035] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260811 18:02:31.286083 28027 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260811 18:02:31.286324 28027 raft_consensus.cc:412] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 2 LEADER]: Emulating election...
I20260811 18:02:31.286494 28027 raft_consensus.cc:3032] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 2 LEADER]: Stepping down as leader of term 2
I20260811 18:02:31.286638 28027 raft_consensus.cc:729] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0c80a2721b0141958cdb190fcfe04e1c, State: Running, Role: LEADER
I20260811 18:02:31.287010 28027 consensus_queue.cc:260] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 } } peers { permanent_uuid: "0c80a2721b0141958cdb190fcfe04e1c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 44143 } }
I20260811 18:02:31.287652 28027 raft_consensus.cc:3037] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:31.292382 28027 raft_consensus.cc:686] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 3 LEADER]: Becoming Leader. State: Replica: 0c80a2721b0141958cdb190fcfe04e1c, State: Running, Role: LEADER
I20260811 18:02:31.293025 28027 consensus_queue.cc:237] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 } } peers { permanent_uuid: "0c80a2721b0141958cdb190fcfe04e1c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 44143 } }
I20260811 18:02:31.294375 28027 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260811 18:02:31.298709 28027 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260811 18:02:31.403775 28027 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260811 18:02:31.610301 28027 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260811 18:02:31.852491 29300 raft_consensus.cc:3037] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:31.857349 29300 raft_consensus.cc:1261] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee [term 3 FOLLOWER]: Refusing update from remote peer 0c80a2721b0141958cdb190fcfe04e1c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260811 18:02:31.859027 29394 consensus_queue.cc:1035] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57b7b81ddf3403cbd1931ec1e63caee" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260811 18:02:31.918144 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:02:31.943652 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:31.944271 28027 tablet_replica.cc:330] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c: stopping tablet replica
I20260811 18:02:31.944801 28027 raft_consensus.cc:2229] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 3 LEADER]: Raft consensus shutting down.
I20260811 18:02:31.945358 28027 raft_consensus.cc:2258] T c7edabe8118e4511a06409a1b420462e P 0c80a2721b0141958cdb190fcfe04e1c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:31.955446 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:02:31.963506 28027 tablet_server.cc:178] TabletServer@127.27.94.194:0 shutting down...
I20260811 18:02:31.981019 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:31.981630 28027 tablet_replica.cc:330] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee: stopping tablet replica
I20260811 18:02:31.982189 28027 raft_consensus.cc:2229] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee [term 3 FOLLOWER]: Raft consensus shutting down.
I20260811 18:02:31.982628 28027 raft_consensus.cc:2258] T c7edabe8118e4511a06409a1b420462e P e57b7b81ddf3403cbd1931ec1e63caee [term 3 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:32.003291 28027 tablet_server.cc:194] TabletServer@127.27.94.194:0 shutdown complete.
I20260811 18:02:32.012619 28027 master.cc:537] Master@127.27.94.254:39917 shutting down...
I20260811 18:02:32.029307 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:02:32.029871 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:32.030189 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P cbb5487d349f41c0a823832332410ba3: stopping tablet replica
I20260811 18:02:32.048928 28027 master.cc:552] Master@127.27.94.254:39917 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2739 ms)
[----------] 3 tests from LeadershipChangeReportingTest (13103 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20260811 18:02:32.078578 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:41431
I20260811 18:02:32.079681 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:32.085183 29409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:32.086333 29411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:32.090977 28027 server_base.cc:1034] running on GCE node
W20260811 18:02:32.090883 29414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:32.092367 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:32.092622 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:32.092794 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471352092778 us; error 0 us; skew 500 ppm
I20260811 18:02:32.093348 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:32.102349 28027 webserver.cc:460] Webserver started at http://127.27.94.254:46349/ using document root <none> and password file <none>
I20260811 18:02:32.102932 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:32.103150 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:32.103428 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:32.104635 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "c2b1b39fb41f47f38013b343174d9ff4"
format_stamp: "Formatted at 2026-08-11 18:02:32 on dist-test-slave-kvfs"
I20260811 18:02:32.113237 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260811 18:02:32.117499 29421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:32.118498 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 18:02:32.118824 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "c2b1b39fb41f47f38013b343174d9ff4"
format_stamp: "Formatted at 2026-08-11 18:02:32 on dist-test-slave-kvfs"
I20260811 18:02:32.119149 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:32.139772 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:32.141306 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:32.189072 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:41431
I20260811 18:02:32.189184 29506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:41431 every 8 connection(s)
I20260811 18:02:32.193832 29507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:32.205439 29507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4: Bootstrap starting.
I20260811 18:02:32.209530 29507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:32.213791 29507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4: No bootstrap required, opened a new log
I20260811 18:02:32.215730 29507 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b1b39fb41f47f38013b343174d9ff4" member_type: VOTER }
I20260811 18:02:32.216243 29507 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:32.216467 29507 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2b1b39fb41f47f38013b343174d9ff4, State: Initialized, Role: FOLLOWER
I20260811 18:02:32.217049 29507 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b1b39fb41f47f38013b343174d9ff4" member_type: VOTER }
I20260811 18:02:32.217538 29507 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:02:32.217777 29507 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:02:32.218052 29507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:32.222731 29507 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b1b39fb41f47f38013b343174d9ff4" member_type: VOTER }
I20260811 18:02:32.223340 29507 leader_election.cc:302] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2b1b39fb41f47f38013b343174d9ff4; no voters:
I20260811 18:02:32.224601 29507 leader_election.cc:288] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:02:32.224932 29511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:02:32.226275 29511 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 1 LEADER]: Becoming Leader. State: Replica: c2b1b39fb41f47f38013b343174d9ff4, State: Running, Role: LEADER
I20260811 18:02:32.226917 29511 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b1b39fb41f47f38013b343174d9ff4" member_type: VOTER }
I20260811 18:02:32.227497 29507 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:02:32.229671 29513 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2b1b39fb41f47f38013b343174d9ff4. Latest consensus state: current_term: 1 leader_uuid: "c2b1b39fb41f47f38013b343174d9ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b1b39fb41f47f38013b343174d9ff4" member_type: VOTER } }
I20260811 18:02:32.229672 29512 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2b1b39fb41f47f38013b343174d9ff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2b1b39fb41f47f38013b343174d9ff4" member_type: VOTER } }
I20260811 18:02:32.230412 29513 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:32.230571 29512 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:32.238451 29518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:02:32.244333 29518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:02:32.245438 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:02:32.254155 29518 catalog_manager.cc:1347] Generated new cluster ID: 1d33df229df14853b5649a948357ae8d
I20260811 18:02:32.254495 29518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:02:32.271596 29518 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:02:32.273366 29518 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:02:32.289016 29518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4: Generated new TSK 0
I20260811 18:02:32.289808 29518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:02:32.311972 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:32.318049 29539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:32.319024 29540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:32.320477 29542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:32.321233 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:32.322059 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:32.322230 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:32.322405 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471352322393 us; error 0 us; skew 500 ppm
I20260811 18:02:32.322984 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:32.325474 28027 webserver.cc:460] Webserver started at http://127.27.94.193:40299/ using document root <none> and password file <none>
I20260811 18:02:32.325966 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:32.326159 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:32.326427 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:32.327651 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "2e8c854d802e4a7fb27bf73b318fbf2c"
format_stamp: "Formatted at 2026-08-11 18:02:32 on dist-test-slave-kvfs"
I20260811 18:02:32.332319 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 18:02:32.335567 29549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:32.336580 28027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260811 18:02:32.336889 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "2e8c854d802e4a7fb27bf73b318fbf2c"
format_stamp: "Formatted at 2026-08-11 18:02:32 on dist-test-slave-kvfs"
I20260811 18:02:32.337215 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:32.351892 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:32.353106 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:32.354574 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:32.358038 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:32.358322 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:32.358608 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:32.358783 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:32.404397 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:33545
I20260811 18:02:32.404497 29663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:33545 every 8 connection(s)
I20260811 18:02:32.409726 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:32.419003 29673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:32.419446 29671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:32.420679 28027 server_base.cc:1034] running on GCE node
W20260811 18:02:32.421432 29675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:32.422894 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:32.423147 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:32.423344 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471352423332 us; error 0 us; skew 500 ppm
I20260811 18:02:32.424108 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:32.426529 29665 heartbeater.cc:347] Connected to a master server at 127.27.94.254:41431
I20260811 18:02:32.426894 29665 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:32.427392 28027 webserver.cc:460] Webserver started at http://127.27.94.194:36691/ using document root <none> and password file <none>
I20260811 18:02:32.427685 29665 heartbeater.cc:511] Master 127.27.94.254:41431 requested a full tablet report, sending...
I20260811 18:02:32.427917 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:32.428184 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:32.428447 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:32.429840 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-1-root/instance:
uuid: "7dfa9152ae6d4d8490df547c8e2c98eb"
format_stamp: "Formatted at 2026-08-11 18:02:32 on dist-test-slave-kvfs"
I20260811 18:02:32.430015 29447 ts_manager.cc:194] Registered new tserver with Master: 2e8c854d802e4a7fb27bf73b318fbf2c (127.27.94.193:33545)
I20260811 18:02:32.432606 29447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60432
I20260811 18:02:32.435631 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 18:02:32.439860 29684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:32.440819 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 18:02:32.441128 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-1-root
uuid: "7dfa9152ae6d4d8490df547c8e2c98eb"
format_stamp: "Formatted at 2026-08-11 18:02:32 on dist-test-slave-kvfs"
I20260811 18:02:32.441425 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:32.469113 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:32.470306 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:32.471837 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:32.474098 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:32.474287 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:32.474493 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:32.474687 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:32.516194 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.194:36815
I20260811 18:02:32.516324 29786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.194:36815 every 8 connection(s)
I20260811 18:02:32.521473 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:32.528663 29792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:32.529587 29793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:32.533175 29797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:32.533308 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:32.534377 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:32.534610 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:32.534838 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471352534820 us; error 0 us; skew 500 ppm
I20260811 18:02:32.535553 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:32.537031 29788 heartbeater.cc:347] Connected to a master server at 127.27.94.254:41431
I20260811 18:02:32.537460 29788 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:32.538224 29788 heartbeater.cc:511] Master 127.27.94.254:41431 requested a full tablet report, sending...
I20260811 18:02:32.538949 28027 webserver.cc:460] Webserver started at http://127.27.94.195:37465/ using document root <none> and password file <none>
I20260811 18:02:32.539601 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:32.539846 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:32.540195 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:32.540570 29447 ts_manager.cc:194] Registered new tserver with Master: 7dfa9152ae6d4d8490df547c8e2c98eb (127.27.94.194:36815)
I20260811 18:02:32.541667 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-2-root/instance:
uuid: "ca2207e68979492884e2507678572092"
format_stamp: "Formatted at 2026-08-11 18:02:32 on dist-test-slave-kvfs"
I20260811 18:02:32.542778 29447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60434
I20260811 18:02:32.547423 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260811 18:02:32.551070 29805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:32.551888 28027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 18:02:32.552263 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-2-root
uuid: "ca2207e68979492884e2507678572092"
format_stamp: "Formatted at 2026-08-11 18:02:32 on dist-test-slave-kvfs"
I20260811 18:02:32.552537 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1786471338663725-28027-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:32.571938 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:32.573235 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:32.574681 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:32.577262 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:32.577450 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:32.577636 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:32.577757 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:32.660559 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.195:39825
I20260811 18:02:32.660547 29897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.195:39825 every 8 connection(s)
I20260811 18:02:32.681577 29899 heartbeater.cc:347] Connected to a master server at 127.27.94.254:41431
I20260811 18:02:32.681989 29899 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:32.682736 29899 heartbeater.cc:511] Master 127.27.94.254:41431 requested a full tablet report, sending...
I20260811 18:02:32.684664 29447 ts_manager.cc:194] Registered new tserver with Master: ca2207e68979492884e2507678572092 (127.27.94.195:39825)
I20260811 18:02:32.684856 28027 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018372346s
I20260811 18:02:32.687198 29447 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60442
I20260811 18:02:32.719190 29447 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260811 18:02:32.720804 29447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 18:02:32.767019 29734 tablet_service.cc:1467] Processing CreateTablet for tablet e18cf2b7c8e9409a8759b1291eac54ba (DEFAULT_TABLE table=test-table [id=dc6c7148b9db48fd872289b6f6b08528]), partition=
I20260811 18:02:32.768365 29734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18cf2b7c8e9409a8759b1291eac54ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:32.774040 29842 tablet_service.cc:1467] Processing CreateTablet for tablet e18cf2b7c8e9409a8759b1291eac54ba (DEFAULT_TABLE table=test-table [id=dc6c7148b9db48fd872289b6f6b08528]), partition=
I20260811 18:02:32.775425 29842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18cf2b7c8e9409a8759b1291eac54ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:32.778112 29601 tablet_service.cc:1467] Processing CreateTablet for tablet e18cf2b7c8e9409a8759b1291eac54ba (DEFAULT_TABLE table=test-table [id=dc6c7148b9db48fd872289b6f6b08528]), partition=
I20260811 18:02:32.779177 29601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18cf2b7c8e9409a8759b1291eac54ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:32.795254 29921 tablet_bootstrap.cc:492] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb: Bootstrap starting.
I20260811 18:02:32.796697 29920 tablet_bootstrap.cc:492] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092: Bootstrap starting.
I20260811 18:02:32.802241 29921 tablet_bootstrap.cc:654] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:32.803653 29923 tablet_bootstrap.cc:492] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c: Bootstrap starting.
I20260811 18:02:32.803908 29920 tablet_bootstrap.cc:654] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:32.810348 29923 tablet_bootstrap.cc:654] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:32.815055 29921 tablet_bootstrap.cc:492] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb: No bootstrap required, opened a new log
I20260811 18:02:32.815055 29920 tablet_bootstrap.cc:492] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092: No bootstrap required, opened a new log
I20260811 18:02:32.815722 29921 ts_tablet_manager.cc:1397] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.005s
I20260811 18:02:32.815811 29920 ts_tablet_manager.cc:1397] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.012s
I20260811 18:02:32.817870 29923 tablet_bootstrap.cc:492] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c: No bootstrap required, opened a new log
I20260811 18:02:32.818293 29923 ts_tablet_manager.cc:1397] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260811 18:02:32.818632 29921 raft_consensus.cc:348] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 } } peers { permanent_uuid: "ca2207e68979492884e2507678572092" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 39825 } } peers { permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 } }
I20260811 18:02:32.819269 29921 raft_consensus.cc:374] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:32.818964 29920 raft_consensus.cc:348] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 } } peers { permanent_uuid: "ca2207e68979492884e2507678572092" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 39825 } } peers { permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 } }
I20260811 18:02:32.819608 29921 raft_consensus.cc:729] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dfa9152ae6d4d8490df547c8e2c98eb, State: Initialized, Role: FOLLOWER
I20260811 18:02:32.819828 29920 raft_consensus.cc:374] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:32.820359 29920 raft_consensus.cc:729] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2207e68979492884e2507678572092, State: Initialized, Role: FOLLOWER
I20260811 18:02:32.820379 29921 consensus_queue.cc:260] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 } } peers { permanent_uuid: "ca2207e68979492884e2507678572092" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 39825 } } peers { permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 } }
I20260811 18:02:32.821027 29923 raft_consensus.cc:348] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 } } peers { permanent_uuid: "ca2207e68979492884e2507678572092" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 39825 } } peers { permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 } }
I20260811 18:02:32.821216 29920 consensus_queue.cc:260] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 } } peers { permanent_uuid: "ca2207e68979492884e2507678572092" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 39825 } } peers { permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 } }
I20260811 18:02:32.821977 29923 raft_consensus.cc:374] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:32.822384 29923 raft_consensus.cc:729] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e8c854d802e4a7fb27bf73b318fbf2c, State: Initialized, Role: FOLLOWER
I20260811 18:02:32.823278 29923 consensus_queue.cc:260] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 } } peers { permanent_uuid: "ca2207e68979492884e2507678572092" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 39825 } } peers { permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 } }
I20260811 18:02:32.826071 29921 ts_tablet_manager.cc:1428] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260811 18:02:32.826447 29788 heartbeater.cc:503] Master 127.27.94.254:41431 was elected leader, sending a full tablet report...
I20260811 18:02:32.829594 29899 heartbeater.cc:503] Master 127.27.94.254:41431 was elected leader, sending a full tablet report...
I20260811 18:02:32.829779 29920 ts_tablet_manager.cc:1428] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092: Time spent starting tablet: real 0.014s user 0.007s sys 0.003s
I20260811 18:02:32.833104 29665 heartbeater.cc:503] Master 127.27.94.254:41431 was elected leader, sending a full tablet report...
I20260811 18:02:32.833313 29923 ts_tablet_manager.cc:1428] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260811 18:02:32.988585 29931 raft_consensus.cc:482] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:32.989315 29931 raft_consensus.cc:504] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 } } peers { permanent_uuid: "ca2207e68979492884e2507678572092" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 39825 } } peers { permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 } }
I20260811 18:02:32.991789 29931 leader_election.cc:288] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7dfa9152ae6d4d8490df547c8e2c98eb (127.27.94.194:36815), 2e8c854d802e4a7fb27bf73b318fbf2c (127.27.94.193:33545)
I20260811 18:02:33.012629 29752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18cf2b7c8e9409a8759b1291eac54ba" candidate_uuid: "ca2207e68979492884e2507678572092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" is_pre_election: true
I20260811 18:02:33.014178 29752 raft_consensus.cc:2454] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca2207e68979492884e2507678572092 in term 0.
I20260811 18:02:33.014542 29619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18cf2b7c8e9409a8759b1291eac54ba" candidate_uuid: "ca2207e68979492884e2507678572092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" is_pre_election: true
I20260811 18:02:33.015479 29619 raft_consensus.cc:2454] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca2207e68979492884e2507678572092 in term 0.
I20260811 18:02:33.016341 29806 leader_election.cc:302] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dfa9152ae6d4d8490df547c8e2c98eb, ca2207e68979492884e2507678572092; no voters:
I20260811 18:02:33.017467 29931 raft_consensus.cc:2781] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 18:02:33.017920 29931 raft_consensus.cc:482] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:02:33.018304 29931 raft_consensus.cc:3037] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:33.028685 29931 raft_consensus.cc:504] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 } } peers { permanent_uuid: "ca2207e68979492884e2507678572092" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 39825 } } peers { permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 } }
I20260811 18:02:33.031069 29931 leader_election.cc:288] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [CANDIDATE]: Term 1 election: Requested vote from peers 7dfa9152ae6d4d8490df547c8e2c98eb (127.27.94.194:36815), 2e8c854d802e4a7fb27bf73b318fbf2c (127.27.94.193:33545)
I20260811 18:02:33.032011 29752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18cf2b7c8e9409a8759b1291eac54ba" candidate_uuid: "ca2207e68979492884e2507678572092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb"
I20260811 18:02:33.032698 29619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18cf2b7c8e9409a8759b1291eac54ba" candidate_uuid: "ca2207e68979492884e2507678572092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c"
I20260811 18:02:33.033007 29752 raft_consensus.cc:3037] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:33.033593 29619 raft_consensus.cc:3037] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:33.042105 29619 raft_consensus.cc:2454] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca2207e68979492884e2507678572092 in term 1.
I20260811 18:02:33.042284 29752 raft_consensus.cc:2454] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca2207e68979492884e2507678572092 in term 1.
I20260811 18:02:33.043421 29806 leader_election.cc:302] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e8c854d802e4a7fb27bf73b318fbf2c, ca2207e68979492884e2507678572092; no voters:
I20260811 18:02:33.044384 29931 raft_consensus.cc:2781] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:02:33.045351 29931 raft_consensus.cc:686] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 1 LEADER]: Becoming Leader. State: Replica: ca2207e68979492884e2507678572092, State: Running, Role: LEADER
I20260811 18:02:33.045991 29931 consensus_queue.cc:237] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 } } peers { permanent_uuid: "ca2207e68979492884e2507678572092" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 39825 } } peers { permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 } }
I20260811 18:02:33.052987 29447 catalog_manager.cc:5495] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca2207e68979492884e2507678572092 (127.27.94.195). New cstate: current_term: 1 leader_uuid: "ca2207e68979492884e2507678572092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca2207e68979492884e2507678572092" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 39825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 } health_report { overall_health: UNKNOWN } } }
I20260811 18:02:33.430828 29931 consensus_queue.cc:1035] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dfa9152ae6d4d8490df547c8e2c98eb" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 36815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 18:02:33.443902 29945 consensus_queue.cc:1035] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e8c854d802e4a7fb27bf73b318fbf2c" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 18:02:34.125422 28027 tablet_replica.cc:330] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c: stopping tablet replica
I20260811 18:02:34.126109 28027 raft_consensus.cc:2229] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 18:02:34.126618 28027 raft_consensus.cc:2258] T e18cf2b7c8e9409a8759b1291eac54ba P 2e8c854d802e4a7fb27bf73b318fbf2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:34.281083 29806 consensus_queue.cc:579] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [LEADER]: The tablet replica hosted on peer 2e8c854d802e4a7fb27bf73b318fbf2c has failed
W20260811 18:02:34.281615 29806 consensus_peers.cc:487] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 -> Peer 2e8c854d802e4a7fb27bf73b318fbf2c (127.27.94.193:33545): Couldn't send request to peer 2e8c854d802e4a7fb27bf73b318fbf2c. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260811 18:02:34.397015 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:02:34.422610 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:34.442914 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:02:34.452800 28027 tablet_server.cc:178] TabletServer@127.27.94.194:0 shutting down...
W20260811 18:02:34.474545 29806 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260811 18:02:34.475986 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:34.476761 28027 tablet_replica.cc:330] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb: stopping tablet replica
I20260811 18:02:34.477469 28027 raft_consensus.cc:2229] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 18:02:34.477908 28027 raft_consensus.cc:2258] T e18cf2b7c8e9409a8759b1291eac54ba P 7dfa9152ae6d4d8490df547c8e2c98eb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260811 18:02:34.478093 29806 consensus_peers.cc:487] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 -> Peer 7dfa9152ae6d4d8490df547c8e2c98eb (127.27.94.194:36815): Couldn't send request to peer 7dfa9152ae6d4d8490df547c8e2c98eb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260811 18:02:34.498378 28027 tablet_server.cc:194] TabletServer@127.27.94.194:0 shutdown complete.
I20260811 18:02:34.507676 28027 tablet_server.cc:178] TabletServer@127.27.94.195:0 shutting down...
I20260811 18:02:34.529608 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:34.530577 28027 tablet_replica.cc:330] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092: stopping tablet replica
I20260811 18:02:34.531347 28027 raft_consensus.cc:2229] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:02:34.532327 28027 raft_consensus.cc:2258] T e18cf2b7c8e9409a8759b1291eac54ba P ca2207e68979492884e2507678572092 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:34.551259 28027 tablet_server.cc:194] TabletServer@127.27.94.195:0 shutdown complete.
I20260811 18:02:34.560702 28027 master.cc:537] Master@127.27.94.254:41431 shutting down...
I20260811 18:02:34.577579 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:02:34.578141 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:34.578429 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P c2b1b39fb41f47f38013b343174d9ff4: stopping tablet replica
I20260811 18:02:34.596975 28027 master.cc:552] Master@127.27.94.254:41431 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (2548 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20260811 18:02:34.627832 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:38379
I20260811 18:02:34.629022 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:34.635600 29962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:34.636674 29964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:34.639952 29968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:34.640009 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:34.640964 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:34.641361 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:34.641494 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471354641484 us; error 0 us; skew 500 ppm
I20260811 18:02:34.641959 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:34.644964 28027 webserver.cc:460] Webserver started at http://127.27.94.254:37729/ using document root <none> and password file <none>
I20260811 18:02:34.645433 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:34.645584 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:34.645802 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:34.647603 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "f488ebe2a16f41cca65b425124807ab9"
format_stamp: "Formatted at 2026-08-11 18:02:34 on dist-test-slave-kvfs"
I20260811 18:02:34.652935 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 18:02:34.656536 29976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:34.657464 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 18:02:34.657779 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "f488ebe2a16f41cca65b425124807ab9"
format_stamp: "Formatted at 2026-08-11 18:02:34 on dist-test-slave-kvfs"
I20260811 18:02:34.658099 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:34.677785 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:34.678910 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:34.726756 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:38379
I20260811 18:02:34.726990 30049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:38379 every 8 connection(s)
I20260811 18:02:34.737315 30050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:34.749248 30050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9: Bootstrap starting.
I20260811 18:02:34.753422 30050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:34.757954 30050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9: No bootstrap required, opened a new log
I20260811 18:02:34.760272 30050 raft_consensus.cc:348] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f488ebe2a16f41cca65b425124807ab9" member_type: VOTER }
I20260811 18:02:34.760658 30050 raft_consensus.cc:374] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:34.760896 30050 raft_consensus.cc:729] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f488ebe2a16f41cca65b425124807ab9, State: Initialized, Role: FOLLOWER
I20260811 18:02:34.761425 30050 consensus_queue.cc:260] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f488ebe2a16f41cca65b425124807ab9" member_type: VOTER }
I20260811 18:02:34.761855 30050 raft_consensus.cc:388] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:02:34.762059 30050 raft_consensus.cc:482] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:02:34.762346 30050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:34.766568 30050 raft_consensus.cc:504] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f488ebe2a16f41cca65b425124807ab9" member_type: VOTER }
I20260811 18:02:34.767089 30050 leader_election.cc:302] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f488ebe2a16f41cca65b425124807ab9; no voters:
I20260811 18:02:34.768496 30050 leader_election.cc:288] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:02:34.769145 30053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:02:34.770533 30053 raft_consensus.cc:686] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 1 LEADER]: Becoming Leader. State: Replica: f488ebe2a16f41cca65b425124807ab9, State: Running, Role: LEADER
I20260811 18:02:34.771178 30053 consensus_queue.cc:237] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f488ebe2a16f41cca65b425124807ab9" member_type: VOTER }
I20260811 18:02:34.771432 30050 sys_catalog.cc:564] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:02:34.774398 30055 sys_catalog.cc:455] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f488ebe2a16f41cca65b425124807ab9. Latest consensus state: current_term: 1 leader_uuid: "f488ebe2a16f41cca65b425124807ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f488ebe2a16f41cca65b425124807ab9" member_type: VOTER } }
I20260811 18:02:34.774338 30054 sys_catalog.cc:455] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f488ebe2a16f41cca65b425124807ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f488ebe2a16f41cca65b425124807ab9" member_type: VOTER } }
I20260811 18:02:34.775259 30055 sys_catalog.cc:458] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:34.775468 30054 sys_catalog.cc:458] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:34.786743 30061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:02:34.792527 30061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:02:34.794090 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:02:34.802487 30061 catalog_manager.cc:1347] Generated new cluster ID: 01b7df19a02a4a9089d9c21361cf0f7f
I20260811 18:02:34.802812 30061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:02:34.832015 30061 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:02:34.834312 30061 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:02:34.849392 30061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9: Generated new TSK 0
I20260811 18:02:34.850211 30061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:02:34.861505 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:34.870445 30079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:34.875869 30080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:34.878273 30086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:34.885985 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:34.887122 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:34.887360 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:34.887600 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471354887579 us; error 0 us; skew 500 ppm
I20260811 18:02:34.888585 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:34.892546 28027 webserver.cc:460] Webserver started at http://127.27.94.193:37067/ using document root <none> and password file <none>
I20260811 18:02:34.893277 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:34.893524 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:34.893880 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:34.895833 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "f49037e977b945d7b9677bdf5c3ef8ec"
format_stamp: "Formatted at 2026-08-11 18:02:34 on dist-test-slave-kvfs"
I20260811 18:02:34.903313 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 18:02:34.909888 30093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:34.912986 28027 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260811 18:02:34.913475 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "f49037e977b945d7b9677bdf5c3ef8ec"
format_stamp: "Formatted at 2026-08-11 18:02:34 on dist-test-slave-kvfs"
I20260811 18:02:34.914153 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:34.937945 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:34.939355 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:34.941519 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:34.945240 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:34.945513 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:34.945806 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:34.945971 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:34.999320 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:41773
I20260811 18:02:34.999433 30202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:41773 every 8 connection(s)
I20260811 18:02:35.021540 30203 heartbeater.cc:347] Connected to a master server at 127.27.94.254:38379
I20260811 18:02:35.021883 30203 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:35.022754 30203 heartbeater.cc:511] Master 127.27.94.254:38379 requested a full tablet report, sending...
I20260811 18:02:35.025014 29999 ts_manager.cc:194] Registered new tserver with Master: f49037e977b945d7b9677bdf5c3ef8ec (127.27.94.193:41773)
I20260811 18:02:35.025221 28027 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.022705175s
I20260811 18:02:35.027220 29999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40102
I20260811 18:02:35.057914 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:02:35.085501 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:35.104300 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:02:35.116649 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 18:02:35.119719 28027 tablet_server.cc:93] deduplicated master addresses: 127.27.94.254:38379
W20260811 18:02:35.124807 30219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:35.125829 30221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:35.129338 30227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:35.130589 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:35.131729 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:35.131935 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:35.132462 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471355132440 us; error 0 us; skew 500 ppm
I20260811 18:02:35.133139 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:35.136117 28027 webserver.cc:460] Webserver started at http://127.27.94.193:37067/ using document root <none> and password file <none>
I20260811 18:02:35.136673 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:35.136827 28027 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260811 18:02:35.140658 28027 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 18:02:35.145244 30233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:35.146490 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260811 18:02:35.146957 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "f49037e977b945d7b9677bdf5c3ef8ec"
format_stamp: "Formatted at 2026-08-11 18:02:34 on dist-test-slave-kvfs"
I20260811 18:02:35.148200 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:35.167348 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:35.169126 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:35.171051 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:35.175627 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:35.175887 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:35.176182 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:35.176354 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:35.242959 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:41773
I20260811 18:02:35.243084 30337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:41773 every 8 connection(s)
I20260811 18:02:35.256445 28027 tablet_server.cc:178] TabletServer@127.27.94.193:41773 shutting down...
I20260811 18:02:35.278795 30338 heartbeater.cc:347] Connected to a master server at 127.27.94.254:38379
I20260811 18:02:35.279338 30338 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:35.280345 30338 heartbeater.cc:511] Master 127.27.94.254:38379 requested a full tablet report, sending...
I20260811 18:02:35.286378 29999 ts_manager.cc:194] Re-registered known tserver with Master: f49037e977b945d7b9677bdf5c3ef8ec (127.27.94.193:41773)
I20260811 18:02:35.289593 29999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40130
I20260811 18:02:35.294222 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:02:35.313009 28027 tablet_server.cc:194] TabletServer@127.27.94.193:41773 shutdown complete.
I20260811 18:02:35.324841 28027 master.cc:537] Master@127.27.94.254:38379 shutting down...
I20260811 18:02:35.348127 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:02:35.348667 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:02:35.349092 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P f488ebe2a16f41cca65b425124807ab9: stopping tablet replica
I20260811 18:02:35.369815 28027 master.cc:552] Master@127.27.94.254:38379 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (769 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20260811 18:02:35.397787 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:42193,127.27.94.253:34173,127.27.94.252:40527
I20260811 18:02:35.399185 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:02:35.405674 30349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:35.406525 30353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:35.408658 30355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:35.409370 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:35.410018 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:35.410194 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:35.410503 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471355410483 us; error 0 us; skew 500 ppm
I20260811 18:02:35.411008 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:35.414480 28027 webserver.cc:460] Webserver started at http://127.27.94.254:35605/ using document root <none> and password file <none>
I20260811 18:02:35.415205 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:35.415491 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:35.416687 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:35.418669 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "baad4df5e5d244749b5730df044077a4"
format_stamp: "Formatted at 2026-08-11 18:02:35 on dist-test-slave-kvfs"
I20260811 18:02:35.426446 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260811 18:02:35.430717 30361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:35.431567 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 18:02:35.431879 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "baad4df5e5d244749b5730df044077a4"
format_stamp: "Formatted at 2026-08-11 18:02:35 on dist-test-slave-kvfs"
I20260811 18:02:35.432205 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:35.477430 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:35.478701 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:35.526475 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:42193
I20260811 18:02:35.526611 30447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:42193 every 8 connection(s)
I20260811 18:02:35.530990 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 18:02:35.531160 30448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20260811 18:02:35.539415 30450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:35.541239 30452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:35.545735 30454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:35.546363 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:35.546060 30448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } has no permanent_uuid. Determining permanent_uuid...
I20260811 18:02:35.547331 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:35.547578 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:35.547760 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471355547747 us; error 0 us; skew 500 ppm
I20260811 18:02:35.548579 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:35.552517 28027 webserver.cc:460] Webserver started at http://127.27.94.253:46301/ using document root <none> and password file <none>
I20260811 18:02:35.553146 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:35.553349 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:35.553663 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:35.555683 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-1-root/instance:
uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb"
format_stamp: "Formatted at 2026-08-11 18:02:35 on dist-test-slave-kvfs"
I20260811 18:02:35.564343 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260811 18:02:35.566911 30448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
I20260811 18:02:35.568797 30465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:35.569887 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 18:02:35.570186 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-1-root
uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb"
format_stamp: "Formatted at 2026-08-11 18:02:35 on dist-test-slave-kvfs"
I20260811 18:02:35.570477 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260811 18:02:35.572613 30448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.94.253:34173: Network error: Client connection negotiation failed: client connection to 127.27.94.253:34173: connect: Connection refused (error 111)
I20260811 18:02:35.598022 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:35.599272 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:35.624342 30448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } attempt: 1
W20260811 18:02:35.629905 30448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.94.253:34173: Network error: Client connection negotiation failed: client connection to 127.27.94.253:34173: connect: Connection refused (error 111)
I20260811 18:02:35.650707 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.253:34173
I20260811 18:02:35.650974 30543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.253:34173 every 8 connection(s)
I20260811 18:02:35.655406 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 18:02:35.656206 30545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20260811 18:02:35.664466 30547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:35.666791 30545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } has no permanent_uuid. Determining permanent_uuid...
W20260811 18:02:35.667495 30552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:35.668872 30557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:35.670228 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:35.671412 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:35.671643 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:35.671821 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471355671807 us; error 0 us; skew 500 ppm
I20260811 18:02:35.672425 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:35.675714 28027 webserver.cc:460] Webserver started at http://127.27.94.252:44527/ using document root <none> and password file <none>
I20260811 18:02:35.676390 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:35.676762 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:35.677075 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:35.678716 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-2-root/instance:
uuid: "ceb28f49354743839ed45f09a79b5b5c"
format_stamp: "Formatted at 2026-08-11 18:02:35 on dist-test-slave-kvfs"
I20260811 18:02:35.687753 30545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
I20260811 18:02:35.687757 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260811 18:02:35.691493 30448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } attempt: 2
I20260811 18:02:35.695485 30573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:35.698292 28027 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260811 18:02:35.698673 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-2-root
uuid: "ceb28f49354743839ed45f09a79b5b5c"
format_stamp: "Formatted at 2026-08-11 18:02:35 on dist-test-slave-kvfs"
I20260811 18:02:35.698988 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:35.712646 30448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260811 18:02:35.712666 30545 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260811 18:02:35.716138 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:35.717803 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260811 18:02:35.720785 30448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.94.252:40527: Network error: Client connection negotiation failed: client connection to 127.27.94.252:40527: connect: Connection refused (error 111)
W20260811 18:02:35.721019 30545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.94.252:40527: Network error: Client connection negotiation failed: client connection to 127.27.94.252:40527: connect: Connection refused (error 111)
I20260811 18:02:35.739334 30448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } attempt: 1
W20260811 18:02:35.747381 30448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.94.252:40527: Network error: Client connection negotiation failed: client connection to 127.27.94.252:40527: connect: Connection refused (error 111)
I20260811 18:02:35.759760 30545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } attempt: 1
W20260811 18:02:35.766983 30545 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.94.252:40527: Network error: Client connection negotiation failed: client connection to 127.27.94.252:40527: connect: Connection refused (error 111)
I20260811 18:02:35.774665 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.252:40527
I20260811 18:02:35.774760 30657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.252:40527 every 8 connection(s)
I20260811 18:02:35.778290 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:02:35.779699 30658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:35.787158 30658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } has no permanent_uuid. Determining permanent_uuid...
I20260811 18:02:35.787997 30448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } attempt: 2
I20260811 18:02:35.812361 30658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } has no permanent_uuid. Determining permanent_uuid...
I20260811 18:02:35.817291 30448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4: Bootstrap starting.
I20260811 18:02:35.818789 30545 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } attempt: 2
I20260811 18:02:35.824780 30448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:35.827816 30658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
I20260811 18:02:35.832201 30448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4: No bootstrap required, opened a new log
I20260811 18:02:35.835000 30448 raft_consensus.cc:348] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:35.835958 30448 raft_consensus.cc:374] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:35.836525 30448 raft_consensus.cc:729] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baad4df5e5d244749b5730df044077a4, State: Initialized, Role: FOLLOWER
I20260811 18:02:35.837743 30448 consensus_queue.cc:260] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:35.839927 30668 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:35.841238 30668 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:35.842115 30448 sys_catalog.cc:564] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:02:35.844807 30545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb: Bootstrap starting.
I20260811 18:02:35.851837 30545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:35.858405 30658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c: Bootstrap starting.
I20260811 18:02:35.858845 30545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb: No bootstrap required, opened a new log
I20260811 18:02:35.861826 30545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:35.862721 30545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:35.863169 30545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5295d3ba6a4b4a5ba4ed3e13a28771fb, State: Initialized, Role: FOLLOWER
I20260811 18:02:35.865134 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20260811 18:02:35.864425 30545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:35.865545 30658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:35.867187 30693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:35.868198 30693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:35.869369 30545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: configured and running, proceeding with master startup.
W20260811 18:02:35.870057 30688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 18:02:35.870375 30688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 18:02:35.872133 30658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c: No bootstrap required, opened a new log
I20260811 18:02:35.875563 30658 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:35.876700 30658 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:35.877077 30658 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb28f49354743839ed45f09a79b5b5c, State: Initialized, Role: FOLLOWER
I20260811 18:02:35.877903 30658 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:35.880196 30713 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:35.881420 30713 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:35.883833 30658 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:02:35.886111 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20260811 18:02:35.889659 30724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 18:02:35.890019 30724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 18:02:35.896646 30713 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:35.897300 30713 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:35.901337 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4" is_pre_election: true
I20260811 18:02:35.902048 30406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb28f49354743839ed45f09a79b5b5c in term 0.
I20260811 18:02:35.902781 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 18:02:35.903250 30583 leader_election.cc:302] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baad4df5e5d244749b5730df044077a4, ceb28f49354743839ed45f09a79b5b5c; no voters:
I20260811 18:02:35.904518 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
W20260811 18:02:35.905669 30743 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260811 18:02:35.905539 30506 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb28f49354743839ed45f09a79b5b5c in term 0.
W20260811 18:02:35.906005 30743 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 18:02:35.907015 30713 leader_election.cc:288] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173)
I20260811 18:02:35.907903 30713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 18:02:35.908459 30713 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:02:35.908840 30713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:35.914651 30713 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
W20260811 18:02:35.916440 30746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:35.918085 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4"
I20260811 18:02:35.918758 30406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:35.919313 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb"
I20260811 18:02:35.919934 30506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:35.924537 30713 leader_election.cc:288] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 1 election: Requested vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173)
W20260811 18:02:35.924783 30747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:35.925153 30751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:35.926894 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:35.926888 30406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb28f49354743839ed45f09a79b5b5c in term 1.
I20260811 18:02:35.927206 30506 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb28f49354743839ed45f09a79b5b5c in term 1.
I20260811 18:02:35.928148 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:35.928445 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:35.928365 30577 leader_election.cc:302] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, ceb28f49354743839ed45f09a79b5b5c; no voters:
I20260811 18:02:35.928751 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471355928735 us; error 0 us; skew 500 ppm
I20260811 18:02:35.929515 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:35.929388 30713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:02:35.930001 30713 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 1 LEADER]: Becoming Leader. State: Replica: ceb28f49354743839ed45f09a79b5b5c, State: Running, Role: LEADER
I20260811 18:02:35.930943 30713 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:35.933123 28027 webserver.cc:460] Webserver started at http://127.27.94.193:35679/ using document root <none> and password file <none>
I20260811 18:02:35.933851 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:35.934094 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:35.934477 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:35.935920 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "f89e2ed2b23e4472a118ef4746b164df"
format_stamp: "Formatted at 2026-08-11 18:02:35 on dist-test-slave-kvfs"
I20260811 18:02:35.936388 30744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb28f49354743839ed45f09a79b5b5c. Latest consensus state: current_term: 1 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:35.937891 30744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:35.939492 30757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:02:35.942996 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 18:02:35.945605 30757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:02:35.948256 30759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:35.949479 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 18:02:35.949877 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "f89e2ed2b23e4472a118ef4746b164df"
format_stamp: "Formatted at 2026-08-11 18:02:35 on dist-test-slave-kvfs"
I20260811 18:02:35.950302 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:35.956082 30506 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Refusing update from remote peer ceb28f49354743839ed45f09a79b5b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 18:02:35.957790 30406 raft_consensus.cc:1261] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Refusing update from remote peer ceb28f49354743839ed45f09a79b5b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 18:02:35.958413 30744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 18:02:35.959329 30713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 18:02:35.978912 30668 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb28f49354743839ed45f09a79b5b5c. Latest consensus state: current_term: 1 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:35.978912 30693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb28f49354743839ed45f09a79b5b5c. Latest consensus state: current_term: 1 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:35.980685 30668 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:35.981009 30693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:35.985191 30760 mvcc.cc:205] Tried to move back new op lower bound from 7317386673981952000 to 7317386673903726592. Current Snapshot: MvccSnapshot[applied={T|T < 7317386673981952000}]
I20260811 18:02:35.987392 30744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:35.989487 30757 catalog_manager.cc:1347] Generated new cluster ID: 768dd58bfe0e482e8fc13c1742ac9359
I20260811 18:02:35.988871 30744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:35.989957 30757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:02:35.991036 30744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:35.991923 30744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:35.992322 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:35.994275 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:35.997321 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:36.000887 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:36.001205 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:36.001513 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:36.001734 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:36.009622 30668 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:36.010481 30668 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:36.018991 30693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:36.020076 30693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:36.086553 30757 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:02:36.089977 30757 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:02:36.141898 30757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c: Generated new TSK 0
I20260811 18:02:36.143180 30757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:02:36.179028 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:38821
I20260811 18:02:36.179795 30871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:38821 every 8 connection(s)
I20260811 18:02:36.219390 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 18:02:36.260782 30880 heartbeater.cc:347] Connected to a master server at 127.27.94.252:40527
I20260811 18:02:36.262389 30880 heartbeater.cc:464] Registering TS with master...
W20260811 18:02:36.256976 30888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:02:36.264647 30889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:36.265528 30880 heartbeater.cc:511] Master 127.27.94.252:40527 requested a full tablet report, sending...
I20260811 18:02:36.271663 30606 ts_manager.cc:194] Registered new tserver with Master: f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821)
I20260811 18:02:36.276129 30879 heartbeater.cc:347] Connected to a master server at 127.27.94.253:34173
I20260811 18:02:36.277045 30879 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:36.278370 30606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46104
I20260811 18:02:36.279403 30879 heartbeater.cc:511] Master 127.27.94.253:34173 requested a full tablet report, sending...
W20260811 18:02:36.280432 30891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:36.281320 28027 server_base.cc:1034] running on GCE node
I20260811 18:02:36.283314 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:36.283733 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:36.283927 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471356283914 us; error 0 us; skew 500 ppm
I20260811 18:02:36.284729 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:36.283753 30883 heartbeater.cc:347] Connected to a master server at 127.27.94.254:42193
I20260811 18:02:36.286034 30883 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:36.286310 30491 ts_manager.cc:194] Registered new tserver with Master: f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821)
I20260811 18:02:36.288071 30883 heartbeater.cc:511] Master 127.27.94.254:42193 requested a full tablet report, sending...
I20260811 18:02:36.290127 28027 webserver.cc:460] Webserver started at http://127.27.94.194:34955/ using document root <none> and password file <none>
I20260811 18:02:36.290805 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:36.291049 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:36.291344 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:36.294077 30386 ts_manager.cc:194] Registered new tserver with Master: f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821)
I20260811 18:02:36.295486 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-1-root/instance:
uuid: "90c3072068094afbafbc8e30ae2dc8c9"
format_stamp: "Formatted at 2026-08-11 18:02:36 on dist-test-slave-kvfs"
I20260811 18:02:36.301798 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 18:02:36.307806 30901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:36.309043 28027 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260811 18:02:36.309406 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-1-root
uuid: "90c3072068094afbafbc8e30ae2dc8c9"
format_stamp: "Formatted at 2026-08-11 18:02:36 on dist-test-slave-kvfs"
I20260811 18:02:36.309758 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:36.350651 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:36.353060 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:36.354837 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:36.357767 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:36.358098 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:36.358371 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:36.358575 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:36.393700 30880 heartbeater.cc:503] Master 127.27.94.252:40527 was elected leader, sending a full tablet report...
I20260811 18:02:36.642577 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.194:38115
I20260811 18:02:36.642781 30995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.194:38115 every 8 connection(s)
I20260811 18:02:36.657946 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 18:02:36.679420 30997 heartbeater.cc:347] Connected to a master server at 127.27.94.253:34173
I20260811 18:02:36.680477 30997 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:36.682835 30997 heartbeater.cc:511] Master 127.27.94.253:34173 requested a full tablet report, sending...
I20260811 18:02:36.689383 30491 ts_manager.cc:194] Registered new tserver with Master: 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:36.701947 31001 heartbeater.cc:347] Connected to a master server at 127.27.94.252:40527
I20260811 18:02:36.703267 31001 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:36.705818 31002 heartbeater.cc:347] Connected to a master server at 127.27.94.254:42193
I20260811 18:02:36.706768 31002 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:36.705840 31001 heartbeater.cc:511] Master 127.27.94.252:40527 requested a full tablet report, sending...
W20260811 18:02:36.707283 31011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:36.710021 31002 heartbeater.cc:511] Master 127.27.94.254:42193 requested a full tablet report, sending...
W20260811 18:02:36.711275 31012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:36.717757 30606 ts_manager.cc:194] Registered new tserver with Master: 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:36.718706 30386 ts_manager.cc:194] Registered new tserver with Master: 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:36.723800 30606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46110
I20260811 18:02:36.746223 28027 server_base.cc:1034] running on GCE node
W20260811 18:02:36.748560 31014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:02:36.749635 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:02:36.749866 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:02:36.750046 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471356750006 us; error 0 us; skew 500 ppm
I20260811 18:02:36.750680 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:02:36.753814 28027 webserver.cc:460] Webserver started at http://127.27.94.195:37965/ using document root <none> and password file <none>
I20260811 18:02:36.754335 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:02:36.754544 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:02:36.754823 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:02:36.756383 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-2-root/instance:
uuid: "148f3fa79b0f430fa5178ae41825982c"
format_stamp: "Formatted at 2026-08-11 18:02:36 on dist-test-slave-kvfs"
I20260811 18:02:36.761052 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 18:02:36.764688 31021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:36.765579 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260811 18:02:36.765918 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-2-root
uuid: "148f3fa79b0f430fa5178ae41825982c"
format_stamp: "Formatted at 2026-08-11 18:02:36 on dist-test-slave-kvfs"
I20260811 18:02:36.766206 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:02:36.788851 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:02:36.790254 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:02:36.791615 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:02:36.794763 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:02:36.795003 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:36.795258 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:02:36.795423 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:02:36.833391 31001 heartbeater.cc:503] Master 127.27.94.252:40527 was elected leader, sending a full tablet report...
I20260811 18:02:36.881606 30688 catalog_manager.cc:1259] Loaded cluster ID: 768dd58bfe0e482e8fc13c1742ac9359
I20260811 18:02:36.882040 30688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4: loading cluster ID for follower catalog manager: success
I20260811 18:02:36.915798 30724 catalog_manager.cc:1259] Loaded cluster ID: 768dd58bfe0e482e8fc13c1742ac9359
I20260811 18:02:36.916736 30724 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb: loading cluster ID for follower catalog manager: success
I20260811 18:02:36.921830 30688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4: acquiring CA information for follower catalog manager: success
I20260811 18:02:36.937983 30724 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb: acquiring CA information for follower catalog manager: success
I20260811 18:02:36.939033 30688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 18:02:36.950274 30724 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 18:02:37.145037 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.195:33295
I20260811 18:02:37.145205 31125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.195:33295 every 8 connection(s)
I20260811 18:02:37.225654 31127 heartbeater.cc:347] Connected to a master server at 127.27.94.252:40527
I20260811 18:02:37.226300 31127 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:37.228012 31127 heartbeater.cc:511] Master 127.27.94.252:40527 requested a full tablet report, sending...
I20260811 18:02:37.230042 31126 heartbeater.cc:347] Connected to a master server at 127.27.94.253:34173
I20260811 18:02:37.230155 31128 heartbeater.cc:347] Connected to a master server at 127.27.94.254:42193
I20260811 18:02:37.230669 31126 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:37.230993 31128 heartbeater.cc:464] Registering TS with master...
I20260811 18:02:37.231307 30606 ts_manager.cc:194] Registered new tserver with Master: 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:37.232666 31128 heartbeater.cc:511] Master 127.27.94.254:42193 requested a full tablet report, sending...
I20260811 18:02:37.232647 31126 heartbeater.cc:511] Master 127.27.94.253:34173 requested a full tablet report, sending...
I20260811 18:02:37.234144 30606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46122
I20260811 18:02:37.237679 30490 ts_manager.cc:194] Registered new tserver with Master: 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:37.238479 30386 ts_manager.cc:194] Registered new tserver with Master: 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:37.239113 28027 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.075724336s
I20260811 18:02:37.323319 30606 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20260811 18:02:37.326057 30606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 18:02:37.342056 31127 heartbeater.cc:503] Master 127.27.94.252:40527 was elected leader, sending a full tablet report...
I20260811 18:02:37.606837 30818 tablet_service.cc:1467] Processing CreateTablet for tablet 56c6ebe49c974aea97bdf8c98ca05ec3 (DEFAULT_TABLE table=test-table [id=975e80cdd70e47f781fed1af75d8e92f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:37.608819 30817 tablet_service.cc:1467] Processing CreateTablet for tablet 526e9c9faff74cbb85a95853c233fdb6 (DEFAULT_TABLE table=test-table [id=975e80cdd70e47f781fed1af75d8e92f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:37.610634 30818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c6ebe49c974aea97bdf8c98ca05ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:37.618233 30815 tablet_service.cc:1467] Processing CreateTablet for tablet 4414d0bbf2fc4a5488b033921fe9ff48 (DEFAULT_TABLE table=test-table [id=975e80cdd70e47f781fed1af75d8e92f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:37.621016 30817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526e9c9faff74cbb85a95853c233fdb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:37.628762 30815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4414d0bbf2fc4a5488b033921fe9ff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:37.628350 31068 tablet_service.cc:1467] Processing CreateTablet for tablet 56c6ebe49c974aea97bdf8c98ca05ec3 (DEFAULT_TABLE table=test-table [id=975e80cdd70e47f781fed1af75d8e92f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:37.631815 31066 tablet_service.cc:1467] Processing CreateTablet for tablet 4414d0bbf2fc4a5488b033921fe9ff48 (DEFAULT_TABLE table=test-table [id=975e80cdd70e47f781fed1af75d8e92f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:37.638305 31068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c6ebe49c974aea97bdf8c98ca05ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:37.637177 31067 tablet_service.cc:1467] Processing CreateTablet for tablet 526e9c9faff74cbb85a95853c233fdb6 (DEFAULT_TABLE table=test-table [id=975e80cdd70e47f781fed1af75d8e92f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:37.642848 31066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4414d0bbf2fc4a5488b033921fe9ff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:37.646674 31067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526e9c9faff74cbb85a95853c233fdb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:37.649179 30945 tablet_service.cc:1467] Processing CreateTablet for tablet 56c6ebe49c974aea97bdf8c98ca05ec3 (DEFAULT_TABLE table=test-table [id=975e80cdd70e47f781fed1af75d8e92f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:37.651782 30944 tablet_service.cc:1467] Processing CreateTablet for tablet 526e9c9faff74cbb85a95853c233fdb6 (DEFAULT_TABLE table=test-table [id=975e80cdd70e47f781fed1af75d8e92f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:37.657795 30941 tablet_service.cc:1467] Processing CreateTablet for tablet 4414d0bbf2fc4a5488b033921fe9ff48 (DEFAULT_TABLE table=test-table [id=975e80cdd70e47f781fed1af75d8e92f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260811 18:02:37.662498 30945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56c6ebe49c974aea97bdf8c98ca05ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:37.668648 30941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4414d0bbf2fc4a5488b033921fe9ff48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:37.680416 30944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526e9c9faff74cbb85a95853c233fdb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:02:37.765295 31167 tablet_bootstrap.cc:492] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9: Bootstrap starting.
I20260811 18:02:37.771641 31172 tablet_bootstrap.cc:492] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c: Bootstrap starting.
I20260811 18:02:37.780125 31168 tablet_bootstrap.cc:492] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df: Bootstrap starting.
I20260811 18:02:37.820765 31168 tablet_bootstrap.cc:654] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:37.821188 31167 tablet_bootstrap.cc:654] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:37.822373 31172 tablet_bootstrap.cc:654] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:37.831950 31167 tablet_bootstrap.cc:492] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9: No bootstrap required, opened a new log
I20260811 18:02:37.834066 31172 tablet_bootstrap.cc:492] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c: No bootstrap required, opened a new log
I20260811 18:02:37.834304 31167 ts_tablet_manager.cc:1397] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9: Time spent bootstrapping tablet: real 0.071s user 0.016s sys 0.011s
I20260811 18:02:37.834954 31172 ts_tablet_manager.cc:1397] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c: Time spent bootstrapping tablet: real 0.066s user 0.014s sys 0.009s
I20260811 18:02:37.836148 31168 tablet_bootstrap.cc:492] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df: No bootstrap required, opened a new log
I20260811 18:02:37.837321 31168 ts_tablet_manager.cc:1397] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.019s
I20260811 18:02:37.842617 31167 raft_consensus.cc:348] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:37.843636 31167 raft_consensus.cc:374] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:37.844201 31167 raft_consensus.cc:729] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Initialized, Role: FOLLOWER
I20260811 18:02:37.844333 31172 raft_consensus.cc:348] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:37.845471 31172 raft_consensus.cc:374] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:37.844926 31168 raft_consensus.cc:348] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:37.845916 31172 raft_consensus.cc:729] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148f3fa79b0f430fa5178ae41825982c, State: Initialized, Role: FOLLOWER
I20260811 18:02:37.846050 31168 raft_consensus.cc:374] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:37.846593 31168 raft_consensus.cc:729] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89e2ed2b23e4472a118ef4746b164df, State: Initialized, Role: FOLLOWER
I20260811 18:02:37.846279 31167 consensus_queue.cc:260] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:37.847033 31172 consensus_queue.cc:260] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:37.847839 31168 consensus_queue.cc:260] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:37.859886 31168 ts_tablet_manager.cc:1428] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df: Time spent starting tablet: real 0.022s user 0.008s sys 0.010s
I20260811 18:02:37.862303 31168 tablet_bootstrap.cc:492] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df: Bootstrap starting.
I20260811 18:02:37.881085 31167 ts_tablet_manager.cc:1428] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9: Time spent starting tablet: real 0.046s user 0.012s sys 0.003s
I20260811 18:02:37.896647 31168 tablet_bootstrap.cc:654] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:37.900703 31172 ts_tablet_manager.cc:1428] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c: Time spent starting tablet: real 0.065s user 0.019s sys 0.028s
I20260811 18:02:37.906369 31167 tablet_bootstrap.cc:492] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9: Bootstrap starting.
I20260811 18:02:37.906203 31180 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:37.914215 31172 tablet_bootstrap.cc:492] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c: Bootstrap starting.
I20260811 18:02:37.916003 31179 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:37.917127 31179 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:37.922247 31179 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:37.923569 31181 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:37.908336 31180 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:37.961753 31180 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:37.950973 31181 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:38.013398 31167 tablet_bootstrap.cc:654] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:37.996541 31181 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:38.027981 31172 tablet_bootstrap.cc:654] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:38.080417 31168 tablet_bootstrap.cc:492] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df: No bootstrap required, opened a new log
I20260811 18:02:38.081148 31168 ts_tablet_manager.cc:1397] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df: Time spent bootstrapping tablet: real 0.219s user 0.013s sys 0.031s
I20260811 18:02:38.132893 31168 raft_consensus.cc:348] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.133785 31168 raft_consensus.cc:374] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:38.134119 31168 raft_consensus.cc:729] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89e2ed2b23e4472a118ef4746b164df, State: Initialized, Role: FOLLOWER
I20260811 18:02:38.134845 31168 consensus_queue.cc:260] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.150601 31172 tablet_bootstrap.cc:492] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c: No bootstrap required, opened a new log
I20260811 18:02:38.152452 31172 ts_tablet_manager.cc:1397] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c: Time spent bootstrapping tablet: real 0.243s user 0.019s sys 0.011s
I20260811 18:02:38.152469 31168 ts_tablet_manager.cc:1428] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df: Time spent starting tablet: real 0.071s user 0.003s sys 0.010s
I20260811 18:02:38.155740 31183 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:38.159083 31183 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.160933 31167 tablet_bootstrap.cc:492] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9: No bootstrap required, opened a new log
I20260811 18:02:38.161825 31167 ts_tablet_manager.cc:1397] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9: Time spent bootstrapping tablet: real 0.258s user 0.022s sys 0.011s
I20260811 18:02:38.164580 31168 tablet_bootstrap.cc:492] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df: Bootstrap starting.
I20260811 18:02:38.163290 31183 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:38.167632 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:38.185913 30838 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 0.
I20260811 18:02:38.227624 30903 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:02:38.233675 31179 raft_consensus.cc:2781] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 18:02:38.236094 31179 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:02:38.237078 31179 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Advancing to term 1
W20260811 18:02:38.238557 30903 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.193:38821, user_credentials={real_user=slave}} blocked reactor thread for 49699.6us
I20260811 18:02:38.229913 31167 raft_consensus.cc:348] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.264928 31172 raft_consensus.cc:348] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:38.268713 31167 raft_consensus.cc:374] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:38.271989 31167 raft_consensus.cc:729] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Initialized, Role: FOLLOWER
I20260811 18:02:38.270020 31172 raft_consensus.cc:374] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:38.276965 31172 raft_consensus.cc:729] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148f3fa79b0f430fa5178ae41825982c, State: Initialized, Role: FOLLOWER
I20260811 18:02:38.282156 31172 consensus_queue.cc:260] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:38.279345 31167 consensus_queue.cc:260] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.318588 31172 ts_tablet_manager.cc:1428] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c: Time spent starting tablet: real 0.164s user 0.031s sys 0.020s
I20260811 18:02:38.321383 31172 tablet_bootstrap.cc:492] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c: Bootstrap starting.
I20260811 18:02:38.322276 31168 tablet_bootstrap.cc:654] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:38.341624 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:38.343179 31087 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 0.
I20260811 18:02:38.344404 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:38.345311 30838 raft_consensus.cc:2454] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148f3fa79b0f430fa5178ae41825982c in term 0.
I20260811 18:02:38.359520 31026 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:02:38.362816 31182 raft_consensus.cc:2781] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 18:02:38.363445 31182 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:02:38.364001 31182 raft_consensus.cc:3037] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:38.382093 31179 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:38.388717 31167 ts_tablet_manager.cc:1428] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9: Time spent starting tablet: real 0.226s user 0.018s sys 0.045s
I20260811 18:02:38.395757 31213 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:38.406409 31172 tablet_bootstrap.cc:654] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c: Neither blocks nor log segments found. Creating new log.
I20260811 18:02:38.413241 31179 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 1 election: Requested vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:38.404220 31213 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.414170 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df"
I20260811 18:02:38.416971 30838 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:38.428397 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c"
I20260811 18:02:38.429636 31087 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:38.448632 31167 tablet_bootstrap.cc:492] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9: Bootstrap starting.
I20260811 18:02:38.454299 31182 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.460677 30960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:38.466006 30960 raft_consensus.cc:2379] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f89e2ed2b23e4472a118ef4746b164df in current term 1: Already voted for candidate 90c3072068094afbafbc8e30ae2dc8c9 in this term.
I20260811 18:02:38.468886 31182 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 1 election: Requested vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:38.470204 30837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:38.479832 31087 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 1.
I20260811 18:02:38.482533 30838 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 1.
W20260811 18:02:38.479315 30903 leader_election.cc:341] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821): Illegal state: must be running to vote when last-logged opid is not known
I20260811 18:02:38.476639 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df"
I20260811 18:02:38.480691 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:38.488212 31168 tablet_bootstrap.cc:492] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df: No bootstrap required, opened a new log
I20260811 18:02:38.558495 31168 ts_tablet_manager.cc:1397] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df: Time spent bootstrapping tablet: real 0.400s user 0.023s sys 0.011s
I20260811 18:02:38.558204 30960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:38.486187 30904 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9; no voters:
W20260811 18:02:38.560906 30904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.195:33295, user_credentials={real_user=slave}} blocked reactor thread for 77803.8us
I20260811 18:02:38.562654 31179 raft_consensus.cc:2781] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:02:38.563485 31179 raft_consensus.cc:686] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 LEADER]: Becoming Leader. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Running, Role: LEADER
I20260811 18:02:38.484194 31213 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
W20260811 18:02:38.555873 30764 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.195:33295, user_credentials={real_user=slave}} blocked reactor thread for 175840us
I20260811 18:02:38.581066 31168 raft_consensus.cc:348] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
W20260811 18:02:38.580781 30903 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.193:38821, user_credentials={real_user=slave}} blocked reactor thread for 89827us
I20260811 18:02:38.580242 31167 tablet_bootstrap.cc:654] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9: Neither blocks nor log segments found. Creating new log.
W20260811 18:02:38.584692 31026 leader_election.cc:341] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115): Illegal state: must be running to vote when last-logged opid is not known
I20260811 18:02:38.590106 30960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9"
I20260811 18:02:38.583529 31168 raft_consensus.cc:374] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:38.568856 30836 raft_consensus.cc:3037] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:38.591466 31168 raft_consensus.cc:729] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89e2ed2b23e4472a118ef4746b164df, State: Initialized, Role: FOLLOWER
W20260811 18:02:38.601115 31026 leader_election.cc:341] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115): Illegal state: must be running to vote when last-logged opid is not known
W20260811 18:02:38.601145 30762 leader_election.cc:341] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115): Illegal state: must be running to vote when last-logged opid is not known
I20260811 18:02:38.595369 31179 consensus_queue.cc:237] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:38.603684 31168 consensus_queue.cc:260] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.625255 31172 tablet_bootstrap.cc:492] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c: No bootstrap required, opened a new log
I20260811 18:02:38.626008 31172 ts_tablet_manager.cc:1397] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c: Time spent bootstrapping tablet: real 0.305s user 0.013s sys 0.031s
I20260811 18:02:38.631033 31168 ts_tablet_manager.cc:1428] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df: Time spent starting tablet: real 0.072s user 0.005s sys 0.009s
I20260811 18:02:38.635846 31172 raft_consensus.cc:348] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.638933 31172 raft_consensus.cc:374] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:38.636626 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:38.636397 31216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:02:38.639467 31172 raft_consensus.cc:729] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148f3fa79b0f430fa5178ae41825982c, State: Initialized, Role: FOLLOWER
I20260811 18:02:38.641510 31085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:38.640558 31216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:38.641541 31183 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:38.647161 31172 consensus_queue.cc:260] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.648974 31183 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.653537 31085 raft_consensus.cc:2379] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f89e2ed2b23e4472a118ef4746b164df in current term 1: Already voted for candidate 148f3fa79b0f430fa5178ae41825982c in this term.
I20260811 18:02:38.652015 31183 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:38.652848 31086 raft_consensus.cc:2379] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f89e2ed2b23e4472a118ef4746b164df in current term 1: Already voted for candidate 90c3072068094afbafbc8e30ae2dc8c9 in this term.
I20260811 18:02:38.659330 31216 leader_election.cc:288] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:02:38.659814 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:38.660343 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:38.657862 31172 ts_tablet_manager.cc:1428] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c: Time spent starting tablet: real 0.031s user 0.006s sys 0.012s
I20260811 18:02:38.661095 31182 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:38.662480 31182 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.664656 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:02:38.667835 31086 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f89e2ed2b23e4472a118ef4746b164df in term 0.
I20260811 18:02:38.662058 30959 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f89e2ed2b23e4472a118ef4746b164df in term 0.
I20260811 18:02:38.672142 31182 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:38.673185 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:38.676641 30762 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:02:38.677578 30764 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:38.675460 30838 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148f3fa79b0f430fa5178ae41825982c in term 0.
I20260811 18:02:38.681183 31026 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:02:38.679417 31183 raft_consensus.cc:2781] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 18:02:38.682376 31182 raft_consensus.cc:2781] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 18:02:38.684218 31182 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:02:38.684638 31182 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:38.685369 30764 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:38.686473 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:38.687208 30959 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148f3fa79b0f430fa5178ae41825982c in term 0.
I20260811 18:02:38.699877 31167 tablet_bootstrap.cc:492] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9: No bootstrap required, opened a new log
I20260811 18:02:38.701539 31167 ts_tablet_manager.cc:1397] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9: Time spent bootstrapping tablet: real 0.253s user 0.007s sys 0.028s
I20260811 18:02:38.706347 31179 consensus_queue.cc:1035] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 18:02:38.708386 31182 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.716653 30836 raft_consensus.cc:2454] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148f3fa79b0f430fa5178ae41825982c in term 1.
I20260811 18:02:38.720340 30837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df"
W20260811 18:02:38.720829 30836 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41026 (request call id 2) took 249 ms (client timeout 240 ms). Trace:
I20260811 18:02:38.721312 30837 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 0 FOLLOWER]: Advancing to term 1
W20260811 18:02:38.721417 30836 rpcz_store.cc:269] 0811 18:02:38.471134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 18:02:38.471305 (+ 171us) service_pool.cc:227] Handling call
0811 18:02:38.720759 (+249454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260811 18:02:38.725766 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
W20260811 18:02:38.727257 31087 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51188 (request call id 3) took 241 ms (client timeout 266 ms). Trace:
W20260811 18:02:38.727980 31087 rpcz_store.cc:269] 0811 18:02:38.486222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 18:02:38.486410 (+ 188us) service_pool.cc:227] Handling call
0811 18:02:38.725507 (+239097us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b58001bd598. stack: 00007fa6622b5033 00007fa6622b4f30 00007fa661c13534 00000000003b4464 00000000003b43f9 00007fa66212bf11 00007fa6622af851 00007fa6622af702 00007fa66b19fb09 00007fa66ace0730 00007fa66ada679b 00007fa66ada6821 00007fa66ad561b0 00007fa66ad55a94 00007fa6686ddc04 00007fa6686ddb92
0811 18:02:38.727132 (+ 1625us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3648384}
I20260811 18:02:38.741169 31182 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 1 election: Requested vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:38.741427 30904 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9; no voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:38.746069 31213 raft_consensus.cc:2726] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 18:02:38.741433 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df"
I20260811 18:02:38.748389 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9"
I20260811 18:02:38.757525 30959 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:38.765825 31228 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:38.775578 31228 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:38.769107 31222 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 18:02:38.771176 31167 raft_consensus.cc:348] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.794051 31167 raft_consensus.cc:374] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:02:38.794744 31167 raft_consensus.cc:729] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Initialized, Role: FOLLOWER
I20260811 18:02:38.802016 30836 raft_consensus.cc:2362] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 148f3fa79b0f430fa5178ae41825982c in term 1. Re-sending same reply.
I20260811 18:02:38.804849 31167 consensus_queue.cc:260] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.818671 31224 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 18:02:38.817433 30837 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148f3fa79b0f430fa5178ae41825982c in term 1.
I20260811 18:02:38.826002 31026 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:02:38.827513 31183 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:02:38.828085 31183 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:38.827514 31182 raft_consensus.cc:2781] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:02:38.845834 31228 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:38.851212 30960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:38.869347 30959 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148f3fa79b0f430fa5178ae41825982c in term 1.
I20260811 18:02:38.870488 30837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:38.867945 31179 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:38.872236 31179 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.880678 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:38.882845 31086 raft_consensus.cc:2379] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90c3072068094afbafbc8e30ae2dc8c9 in current term 1: Already voted for candidate 148f3fa79b0f430fa5178ae41825982c in this term.
I20260811 18:02:38.888299 31026 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c; no voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:38.891106 31228 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:38.896507 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4" is_pre_election: true
I20260811 18:02:38.895779 31183 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.904192 31026 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df; no voters: 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:38.906738 31228 raft_consensus.cc:2781] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:02:38.907683 31228 raft_consensus.cc:686] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 1 LEADER]: Becoming Leader. State: Replica: 148f3fa79b0f430fa5178ae41825982c, State: Running, Role: LEADER
I20260811 18:02:38.926805 31182 raft_consensus.cc:686] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 1 LEADER]: Becoming Leader. State: Replica: 148f3fa79b0f430fa5178ae41825982c, State: Running, Role: LEADER
I20260811 18:02:38.887471 31179 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:38.933987 31182 consensus_queue.cc:237] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.927168 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:38.972714 31167 ts_tablet_manager.cc:1428] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9: Time spent starting tablet: real 0.270s user 0.006s sys 0.015s
I20260811 18:02:38.979346 30838 raft_consensus.cc:2379] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90c3072068094afbafbc8e30ae2dc8c9 in current term 1: Already voted for candidate 148f3fa79b0f430fa5178ae41825982c in this term.
I20260811 18:02:38.962239 31228 consensus_queue.cc:237] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:38.968518 31179 consensus_queue.cc:1035] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260811 18:02:38.994444 31183 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 election: Requested vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:39.009279 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9"
I20260811 18:02:39.010324 30959 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:39.103087 31245 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:02:39.105239 31245 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:39.108494 31245 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
W20260811 18:02:40.019008 30364 leader_election.cc:334] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527): Timed out: RequestConsensusVote RPC to 127.27.94.252:40527 timed out after 0.247s (SENT)
W20260811 18:02:40.073822 30362 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.003s sys 0.016s
W20260811 18:02:40.074805 30362 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.672s user 0.003s sys 0.016s
W20260811 18:02:40.100322 30441 debug-util.cc:398] Leaking SignalData structure 0x7b080015df00 after lost signal to thread 28030
W20260811 18:02:40.075626 30362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.253:34173, user_credentials={real_user=slave}} blocked reactor thread for 673063us
W20260811 18:02:40.332238 30762 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.27.94.194:38115 timed out after 0.295s (SENT) [suppressed 17 similar messages]
W20260811 18:02:40.379822 30441 debug-util.cc:398] Leaking SignalData structure 0x7b0800053020 after lost signal to thread 30447
W20260811 18:02:40.511401 30362 leader_election.cc:334] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173): Timed out: RequestConsensusVote RPC to 127.27.94.253:34173 timed out after 0.247s (SENT)
I20260811 18:02:40.512005 30362 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: baad4df5e5d244749b5730df044077a4; no voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, ceb28f49354743839ed45f09a79b5b5c
I20260811 18:02:40.587678 31254 raft_consensus.cc:2726] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 18:02:40.616060 30362 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.253:34173, user_credentials={real_user=slave}} blocked reactor thread for 104690us
W20260811 18:02:40.639961 30762 leader_election.cc:334] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115): Timed out: RequestConsensusVote RPC to 127.27.94.194:38115 timed out after 0.295s (SENT)
I20260811 18:02:40.656564 31256 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:40.657544 31256 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:40.775807 31256 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:40.847920 31254 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:02:40.848804 31254 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:40.852348 31254 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
W20260811 18:02:40.986034 30764 leader_election.cc:334] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295): Timed out: RequestConsensusVote RPC to 127.27.94.195:33295 timed out after 0.295s (SENT)
I20260811 18:02:40.986699 30764 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
W20260811 18:02:41.146483 30364 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.252:40527, user_credentials={real_user=slave}} blocked reactor thread for 62876.4us
W20260811 18:02:41.385394 30364 leader_election.cc:334] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527): Timed out: RequestConsensusVote RPC to 127.27.94.252:40527 timed out after 0.231s (SENT)
W20260811 18:02:41.422329 30764 leader_election.cc:334] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295): Timed out: RequestConsensusVote RPC to 127.27.94.195:33295 timed out after 0.322s (SENT)
W20260811 18:02:41.423517 30762 leader_election.cc:334] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115): Timed out: RequestConsensusVote RPC to 127.27.94.194:38115 timed out after 0.322s (SENT)
I20260811 18:02:41.424124 30762 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:41.520695 31183 raft_consensus.cc:2726] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260811 18:02:41.521677 31183 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260811 18:02:41.566056 30762 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.194:38115, user_credentials={real_user=slave}} blocked reactor thread for 142574us
I20260811 18:02:41.574100 30503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
I20260811 18:02:41.576277 30503 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baad4df5e5d244749b5730df044077a4 in term 1.
I20260811 18:02:41.603753 30958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:41.604229 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:41.604815 30958 raft_consensus.cc:2454] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f89e2ed2b23e4472a118ef4746b164df in term 0.
W20260811 18:02:41.612607 31087 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.542s user 0.011s sys 0.028s
W20260811 18:02:41.613276 31087 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.543s user 0.011s sys 0.028s
I20260811 18:02:41.614635 31256 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:41.615278 31256 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:41.616904 30836 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41012 (request call id 6) took 2680 ms. Trace:
I20260811 18:02:41.617365 30836 rpcz_store.cc:276] 0811 18:02:38.936797 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 18:02:38.937005 (+ 208us) service_pool.cc:227] Handling call
0811 18:02:38.937980 (+ 975us) raft_consensus.cc:1457] Updating replica for 0 ops
0811 18:02:38.938398 (+ 418us) raft_consensus.cc:1520] Early marking committed up to index 1
0811 18:02:39.004338 (+ 65940us) log.cc:842] Serialized 14 byte log entry
0811 18:02:41.614205 (+2609867us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0811 18:02:41.614334 (+ 129us) raft_consensus.cc:1631] Marking committed up to 1
0811 18:02:41.614433 (+ 99us) raft_consensus.cc:1683] Filling consensus response to leader.
0811 18:02:41.616397 (+ 1964us) raft_consensus.cc:1677] UpdateReplicas() finished
0811 18:02:41.616636 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11150,"prepare.queue_time_us":93912,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":715,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":1147,"thread_start_us":56645,"threads_started":1}
I20260811 18:02:41.622217 31087 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51188 (request call id 6) took 2606 ms. Trace:
I20260811 18:02:41.622634 31087 rpcz_store.cc:276] 0811 18:02:39.016147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 18:02:39.016341 (+ 194us) service_pool.cc:227] Handling call
0811 18:02:39.017010 (+ 669us) raft_consensus.cc:1457] Updating replica for 1 ops
0811 18:02:39.066370 (+ 49360us) raft_consensus.cc:1520] Early marking committed up to index 0
0811 18:02:39.066731 (+ 361us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0811 18:02:39.069632 (+ 2901us) log.cc:842] Serialized 26 byte log entry
0811 18:02:41.613952 (+2544320us) raft_consensus.cc:1631] Marking committed up to 1
0811 18:02:41.618536 (+ 4584us) log.cc:842] Serialized 14 byte log entry
0811 18:02:41.621558 (+ 3022us) raft_consensus.cc:1683] Filling consensus response to leader.
0811 18:02:41.621735 (+ 177us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0811 18:02:41.621788 (+ 53us) raft_consensus.cc:1671] finished
0811 18:02:41.621814 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0811 18:02:41.622034 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2036,"prepare.queue_time_us":4194,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":25962,"raft.run_cpu_time_us":1025,"raft.run_wall_time_us":21880,"thread_start_us":2585083,"threads_started":5,"wal-append.queue_time_us":39058,"wal-append.run_cpu_time_us":5503,"wal-append.run_wall_time_us":1005397}
I20260811 18:02:41.642014 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
I20260811 18:02:41.643126 30506 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baad4df5e5d244749b5730df044077a4 in term 1.
I20260811 18:02:41.644785 31267 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:41.645637 31267 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:41.647317 31085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:41.648159 31267 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:41.649041 31084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:41.651489 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c"
I20260811 18:02:41.676693 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:41.678001 30836 raft_consensus.cc:2379] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90c3072068094afbafbc8e30ae2dc8c9 in current term 1: Already voted for candidate 148f3fa79b0f430fa5178ae41825982c in this term.
W20260811 18:02:41.681193 30470 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 2.792s user 0.015s sys 0.035s
I20260811 18:02:41.691671 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
W20260811 18:02:41.692386 30470 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.804s user 0.016s sys 0.036s
I20260811 18:02:41.692540 30506 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baad4df5e5d244749b5730df044077a4 in term 1.
W20260811 18:02:41.692914 30470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.254:42193, user_credentials={real_user=slave}} blocked reactor thread for 2.80472e+06us
I20260811 18:02:41.689734 30903 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9; no voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:41.693969 30362 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, baad4df5e5d244749b5730df044077a4; no voters: ceb28f49354743839ed45f09a79b5b5c
I20260811 18:02:41.696360 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4" is_pre_election: true
I20260811 18:02:41.697613 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
I20260811 18:02:41.697774 30406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5295d3ba6a4b4a5ba4ed3e13a28771fb in term 1.
I20260811 18:02:41.698431 30501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baad4df5e5d244749b5730df044077a4 in term 1.
I20260811 18:02:41.699545 31269 raft_consensus.cc:3037] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:02:41.699949 30470 leader_election.cc:302] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, baad4df5e5d244749b5730df044077a4; no voters: ceb28f49354743839ed45f09a79b5b5c
I20260811 18:02:41.709036 31277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 18:02:41.721742 31277 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:02:41.723110 31277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:41.725800 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c"
I20260811 18:02:41.733289 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
W20260811 18:02:41.737537 30441 debug-util.cc:398] Leaking SignalData structure 0x7b0800067c40 after lost signal to thread 30543
I20260811 18:02:41.737255 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:02:41.740475 30838 raft_consensus.cc:1226] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Rejecting Update request from peer 148f3fa79b0f430fa5178ae41825982c for earlier term 1. Current term is 2. Ops: []
I20260811 18:02:41.741189 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:02:41.741798 30838 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41026 (request call id 7) took 1471 ms. Trace:
I20260811 18:02:41.742167 30838 rpcz_store.cc:276] 0811 18:02:40.270165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 18:02:41.739501 (+1469336us) service_pool.cc:227] Handling call
0811 18:02:41.740048 (+ 547us) raft_consensus.cc:1457] Updating replica for 0 ops
0811 18:02:41.741450 (+ 1402us) raft_consensus.cc:1683] Filling consensus response to leader.
0811 18:02:41.741666 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260811 18:02:41.743391 30960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9"
I20260811 18:02:41.744000 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:02:41.712947 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:41.713603 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:41.714967 31256 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
W20260811 18:02:41.739976 30959 consensus_meta.cc:287] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9: Time spent flushing consensus metadata: real 2.729s user 0.022s sys 0.023s
I20260811 18:02:41.712332 31269 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 18:02:41.750165 30959 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f89e2ed2b23e4472a118ef4746b164df in term 2.
W20260811 18:02:41.750924 30959 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35120 (request call id 5) took 2742 ms (client timeout 295 ms). Trace:
W20260811 18:02:41.751616 30959 rpcz_store.cc:269] 0811 18:02:39.008687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 18:02:39.008899 (+ 212us) service_pool.cc:227] Handling call
0811 18:02:41.750779 (+2741880us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260811 18:02:41.750984 31279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 18:02:41.752673 31279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:02:41.753456 31279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 1 FOLLOWER]: Advancing to term 2
W20260811 18:02:41.752758 30470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.254:42193, user_credentials={real_user=slave}} blocked reactor thread for 53038us
W20260811 18:02:41.754792 30441 debug-util.cc:398] Leaking SignalData structure 0x7b0800228880 after lost signal to thread 30657
I20260811 18:02:41.755069 31275 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:41.755867 31275 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
W20260811 18:02:41.756796 30441 debug-util.cc:398] Leaking SignalData structure 0x7b08002294e0 after lost signal to thread 30871
I20260811 18:02:41.757277 31267 consensus_queue.cc:1046] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 18:02:41.760079 31266 raft_consensus.cc:3032] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 1 LEADER]: Stepping down as leader of term 1
I20260811 18:02:41.760716 31266 raft_consensus.cc:729] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 148f3fa79b0f430fa5178ae41825982c, State: Running, Role: LEADER
I20260811 18:02:41.761816 31266 consensus_queue.cc:260] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
W20260811 18:02:41.757227 30441 debug-util.cc:398] Leaking SignalData structure 0x7b0800229300 after lost signal to thread 30880
I20260811 18:02:41.763861 30762 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
W20260811 18:02:41.765239 30441 debug-util.cc:398] Leaking SignalData structure 0x7b0800229280 after lost signal to thread 30995
I20260811 18:02:41.765729 31256 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:41.752652 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:41.766381 30960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:41.768806 30959 raft_consensus.cc:1226] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Rejecting Update request from peer 148f3fa79b0f430fa5178ae41825982c for earlier term 1. Current term is 2. Ops: []
W20260811 18:02:41.765655 30441 debug-util.cc:398] Leaking SignalData structure 0x7b08002296c0 after lost signal to thread 31001
I20260811 18:02:41.768467 30603 catalog_manager.cc:5495] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194). New cstate: current_term: 1 leader_uuid: "90c3072068094afbafbc8e30ae2dc8c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } health_report { overall_health: HEALTHY } } }
I20260811 18:02:41.771476 31267 consensus_queue.cc:1035] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 18:02:41.773310 31267 consensus_queue.cc:1035] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 18:02:41.775415 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:41.778688 31026 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c; no voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:41.780534 31284 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:41.776302 30836 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 2.
I20260811 18:02:41.781605 31275 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
W20260811 18:02:41.775483 30441 debug-util.cc:398] Leaking SignalData structure 0x7b0800244ce0 after lost signal to thread 31125
I20260811 18:02:41.783790 31266 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:41.793804 30903 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:02:41.795603 30606 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46104 (request call id 21) took 3171 ms. Trace:
I20260811 18:02:41.798327 30606 rpcz_store.cc:276] 0811 18:02:38.623998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 18:02:38.624598 (+ 600us) service_pool.cc:227] Handling call
0811 18:02:41.795346 (+3170748us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3124956,"reported_tablets":1}
I20260811 18:02:41.789037 30601 catalog_manager.cc:5495] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c reported cstate change: term changed from 0 to 1, leader changed from <none> to 148f3fa79b0f430fa5178ae41825982c (127.27.94.195). New cstate: current_term: 1 leader_uuid: "148f3fa79b0f430fa5178ae41825982c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } health_report { overall_health: HEALTHY } } }
I20260811 18:02:41.801729 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:41.801919 31277 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:41.805068 30406 raft_consensus.cc:1226] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 2 FOLLOWER]: Rejecting Update request from peer ceb28f49354743839ed45f09a79b5b5c for earlier term 1. Current term is 2. Ops: []
W20260811 18:02:41.805050 30441 debug-util.cc:398] Leaking SignalData structure 0x7b08001ead40 after lost signal to thread 31211
I20260811 18:02:41.806134 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb"
I20260811 18:02:41.808274 31212 raft_consensus.cc:2781] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260811 18:02:41.808749 31212 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:02:41.809134 31212 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:41.826954 31072 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 }, Status: INVALID_TERM, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260811 18:02:41.826449 31277 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 2 election: Requested vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:02:41.838608 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c"
I20260811 18:02:41.839337 31279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:41.842803 30364 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: baad4df5e5d244749b5730df044077a4; no voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, ceb28f49354743839ed45f09a79b5b5c
I20260811 18:02:41.843369 30501 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 2 FOLLOWER]: Rejecting Update request from peer ceb28f49354743839ed45f09a79b5b5c for earlier term 1. Current term is 2. Ops: []
I20260811 18:02:41.844096 31277 raft_consensus.cc:2726] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260811 18:02:41.852507 31087 raft_consensus.cc:2396] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 90c3072068094afbafbc8e30ae2dc8c9 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260811 18:02:41.867419 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4"
I20260811 18:02:41.868331 31279 leader_election.cc:288] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 2 election: Requested vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:02:41.870002 31212 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:41.868656 30406 raft_consensus.cc:2379] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5295d3ba6a4b4a5ba4ed3e13a28771fb in current term 2: Already voted for candidate baad4df5e5d244749b5730df044077a4 in this term.
I20260811 18:02:41.869357 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c"
I20260811 18:02:41.873157 31072 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 }, Status: INVALID_TERM, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260811 18:02:41.879267 30470 leader_election.cc:302] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb; no voters: baad4df5e5d244749b5730df044077a4, ceb28f49354743839ed45f09a79b5b5c
I20260811 18:02:41.880883 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df"
I20260811 18:02:41.881657 30836 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:41.882799 31279 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260811 18:02:41.901458 31212 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 3 election: Requested vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:41.910216 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c"
I20260811 18:02:41.910971 31087 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:41.913772 30836 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 3.
I20260811 18:02:41.915779 30903 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:02:41.916878 31275 raft_consensus.cc:2781] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 18:02:41.917430 31275 raft_consensus.cc:686] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 LEADER]: Becoming Leader. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Running, Role: LEADER
I20260811 18:02:41.918490 31275 consensus_queue.cc:237] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:41.937959 31087 raft_consensus.cc:2396] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 90c3072068094afbafbc8e30ae2dc8c9 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260811 18:02:41.940292 30501 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 2 FOLLOWER]: Rejecting Update request from peer ceb28f49354743839ed45f09a79b5b5c for earlier term 1. Current term is 2. Ops: [1.7-1.7]
I20260811 18:02:41.940989 30406 raft_consensus.cc:1226] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 2 FOLLOWER]: Rejecting Update request from peer ceb28f49354743839ed45f09a79b5b5c for earlier term 1. Current term is 2. Ops: [1.7-1.7]
I20260811 18:02:41.943107 31288 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 }, Status: INVALID_TERM, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260811 18:02:41.947398 31293 raft_consensus.cc:3032] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 1 LEADER]: Stepping down as leader of term 1
I20260811 18:02:41.951440 31293 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ceb28f49354743839ed45f09a79b5b5c, State: Running, Role: LEADER
I20260811 18:02:41.951728 31072 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 }, Status: INVALID_TERM, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260811 18:02:41.953415 31293 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:41.955718 31293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 1 FOLLOWER]: Advancing to term 2
E20260811 18:02:41.967893 30601 catalog_manager.cc:5650] Error updating tablets from 148f3fa79b0f430fa5178ae41825982c: Illegal state: Replica ceb28f49354743839ed45f09a79b5b5c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }. Tablet report was: is_incremental: true updated_tablets { tablet_id: "526e9c9faff74cbb85a95853c233fdb6" state: RUNNING consensus_state { current_term: 1 leader_uuid: "148f3fa79b0f430fa5178ae41825982c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } sequence_number: 13
I20260811 18:02:42.009930 31284 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:42.010669 31284 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:42.013209 31284 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:42.014484 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:42.014616 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:42.015321 30836 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148f3fa79b0f430fa5178ae41825982c in term 3.
I20260811 18:02:42.017077 31026 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df; no voters: 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:42.018217 31284 raft_consensus.cc:2781] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260811 18:02:42.018653 31284 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:02:42.019044 31284 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:02:42.032090 30836 raft_consensus.cc:1261] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Refusing update from remote peer 90c3072068094afbafbc8e30ae2dc8c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch)
I20260811 18:02:42.034512 31275 consensus_queue.cc:1035] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260811 18:02:42.042857 31144 master_proxy_rpc.cc:203] Re-attempting IsCreateTableDone request to leader Master (127.27.94.252:40527)
I20260811 18:02:42.106730 31284 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:42.110180 31087 raft_consensus.cc:1226] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Rejecting Update request from peer 90c3072068094afbafbc8e30ae2dc8c9 for earlier term 3. Current term is 4. Ops: []
I20260811 18:02:42.113164 31212 consensus_queue.cc:1046] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 18:02:42.113996 31212 consensus_queue.cc:1225] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Peer 148f3fa79b0f430fa5178ae41825982c log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260811 18:02:42.114275 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df"
I20260811 18:02:42.114624 31280 raft_consensus.cc:3032] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 LEADER]: Stepping down as leader of term 3
I20260811 18:02:42.115026 31280 raft_consensus.cc:729] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Running, Role: LEADER
I20260811 18:02:42.115922 31280 consensus_queue.cc:260] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.1, Last appended by leader: 1, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:42.117448 31280 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:02:42.129886 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9"
I20260811 18:02:42.131006 30959 raft_consensus.cc:2396] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 148f3fa79b0f430fa5178ae41825982c for term 4 because replica has last-logged OpId of term: 3 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260811 18:02:42.131992 31284 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 4 election: Requested vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:42.132468 31026 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c; no voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:42.133549 31284 raft_consensus.cc:2726] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260811 18:02:42.148478 31293 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:42.149076 31293 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:42.151238 31293 leader_election.cc:288] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173)
I20260811 18:02:42.153209 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
I20260811 18:02:42.153998 30501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb28f49354743839ed45f09a79b5b5c in term 2.
I20260811 18:02:42.155252 30577 leader_election.cc:302] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, ceb28f49354743839ed45f09a79b5b5c; no voters:
I20260811 18:02:42.156275 31293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260811 18:02:42.156755 31293 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:02:42.157213 31293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:42.157480 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4" is_pre_election: true
I20260811 18:02:42.158238 30406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb28f49354743839ed45f09a79b5b5c in term 2.
I20260811 18:02:42.170428 31293 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:42.174391 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4"
I20260811 18:02:42.175283 30406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:42.177241 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb"
I20260811 18:02:42.178025 30501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:42.182474 31293 leader_election.cc:288] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 3 election: Requested vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173)
I20260811 18:02:42.183715 30406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb28f49354743839ed45f09a79b5b5c in term 3.
I20260811 18:02:42.187117 30583 leader_election.cc:302] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baad4df5e5d244749b5730df044077a4, ceb28f49354743839ed45f09a79b5b5c; no voters:
I20260811 18:02:42.205717 31293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 3 FOLLOWER]: Leader election won for term 3
I20260811 18:02:42.206382 30501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb28f49354743839ed45f09a79b5b5c in term 3.
I20260811 18:02:42.215749 31293 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 3 LEADER]: Becoming Leader. State: Replica: ceb28f49354743839ed45f09a79b5b5c, State: Running, Role: LEADER
I20260811 18:02:42.218041 31293 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:42.226163 31072 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb28f49354743839ed45f09a79b5b5c. Latest consensus state: current_term: 3 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:42.228479 31072 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:42.285466 31256 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:42.286002 31256 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:42.287624 31256 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:42.288486 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:42.289163 30959 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f89e2ed2b23e4472a118ef4746b164df in term 4.
I20260811 18:02:42.289239 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:42.290035 31087 raft_consensus.cc:2379] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f89e2ed2b23e4472a118ef4746b164df in current term 4: Already voted for candidate 148f3fa79b0f430fa5178ae41825982c in this term.
I20260811 18:02:42.290495 30762 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:02:42.291636 31256 raft_consensus.cc:2781] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260811 18:02:42.292065 31256 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Starting leader election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:42.292405 31256 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:02:42.306589 31256 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:42.309758 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9"
I20260811 18:02:42.319396 30501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 3 FOLLOWER]: Refusing update from remote peer ceb28f49354743839ed45f09a79b5b5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20260811 18:02:42.324060 30406 raft_consensus.cc:1261] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 3 FOLLOWER]: Refusing update from remote peer ceb28f49354743839ed45f09a79b5b5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20260811 18:02:42.326550 31293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260811 18:02:42.328646 31293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260811 18:02:42.352695 31256 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 4 election: Requested vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:42.352910 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c"
I20260811 18:02:42.349697 30959 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f89e2ed2b23e4472a118ef4746b164df in term 4.
I20260811 18:02:42.353695 31279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb28f49354743839ed45f09a79b5b5c. Latest consensus state: current_term: 3 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:42.354518 31087 raft_consensus.cc:2379] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f89e2ed2b23e4472a118ef4746b164df in current term 4: Already voted for candidate 148f3fa79b0f430fa5178ae41825982c in this term.
I20260811 18:02:42.354590 31279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:42.355401 30762 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:02:42.356789 31256 raft_consensus.cc:2781] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 4 FOLLOWER]: Leader election won for term 4
I20260811 18:02:42.368034 31256 raft_consensus.cc:686] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 4 LEADER]: Becoming Leader. State: Replica: f89e2ed2b23e4472a118ef4746b164df, State: Running, Role: LEADER
I20260811 18:02:42.369268 31256 consensus_queue.cc:237] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:42.376300 31277 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb28f49354743839ed45f09a79b5b5c. Latest consensus state: current_term: 3 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:42.377705 31277 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:42.408068 31279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:42.409427 31279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:42.421000 31284 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:42.421614 31284 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:42.423856 31284 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:42.432396 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:42.434067 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:42.435128 30959 raft_consensus.cc:2396] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148f3fa79b0f430fa5178ae41825982c for term 5 because replica has last-logged OpId of term: 3 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260811 18:02:42.436215 31288 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:42.436501 31072 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:42.437141 31288 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:42.437337 31072 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:42.439113 31026 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c; no voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:42.440685 31284 raft_consensus.cc:2726] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260811 18:02:42.443933 30603 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46110 (request call id 17) took 3594 ms. Trace:
I20260811 18:02:42.459052 31277 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:42.460350 31277 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:42.452960 30603 rpcz_store.cc:276] 0811 18:02:38.849903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 18:02:38.850559 (+ 656us) service_pool.cc:227] Handling call
0811 18:02:42.443856 (+3593297us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0811 18:02:41.827177 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0811 18:02:41.832750 (+ 5573us) write_op.cc:432] Acquiring schema lock in shared mode
0811 18:02:41.832799 (+ 49us) write_op.cc:435] Acquired schema lock
0811 18:02:41.832836 (+ 37us) tablet.cc:660] Decoding operations
0811 18:02:41.833336 (+ 500us) write_op.cc:620] Acquiring the partition lock for write op
0811 18:02:41.833439 (+ 103us) write_op.cc:641] Partition lock acquired for write op
0811 18:02:41.833480 (+ 41us) tablet.cc:683] Acquiring locks for 1 operations
0811 18:02:41.833752 (+ 272us) tablet.cc:699] Row locks acquired
0811 18:02:41.833786 (+ 34us) write_op.cc:260] PREPARE: finished
0811 18:02:41.833948 (+ 162us) write_op.cc:270] Start()
0811 18:02:41.834093 (+ 145us) write_op.cc:276] Timestamp: P: 1786471361833889 usec, L: 0
0811 18:02:41.834137 (+ 44us) op_driver.cc:331] REPLICATION: starting
0811 18:02:41.881815 (+ 47678us) spinlock_profiling.cc:243] Waited 42 ms on lock 0x7b5400124ba0. stack: 00007fa6622b5033 00007fa6622b4f30 00007fa661c13534 00000000003b4464 00000000003b43f9 00007fa668acd471 00007fa668adddc0 00007fa668b4ebd2 00007fa668b442d7 00007fa668b44114 00007fa668fa5ade 00007fa668fa52b8 00007fa668fa9ba1 00007fa668fa9b59 00007fa668fa9ae9 00007fa668fa9ab1
0811 18:02:41.882531 (+ 716us) log.cc:842] Serialized 543 byte log entry
0811 18:02:41.944770 (+ 62239us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b5400124ba0. stack: 00007fa6622b5033 00007fa6622b4f30 00007fa661c13534 00000000003b4464 00000000003b43f9 00007fa668acd471 00007fa668ad6771 00007fa668ae1220 00007fa668ae11c8 00007fa668ae1148 00007fa668ae1103 00007fa668ae038c 00007fa66b218206 00007fa66b210808 00007fa668b1d38b 00007fa668b1fe80
0811 18:02:41.946886 (+ 2116us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b5400124ba0. stack: 00007fa6622b5033 00007fa6622b4f30 00007fa661c13534 00000000003b4464 00000000003b43f9 00007fa668acdaf0 00007fa668ad8028 00007fa668ad7c6a 00007fa668b43cd5 00007fa668b44179 00007fa668fa5ade 00007fa668fa52b8 00007fa668fa9ba1 00007fa668fa9b59 00007fa668fa9ae9 00007fa668fa9ab1
0811 18:02:41.953742 (+ 6856us) spinlock_profiling.cc:243] Waited 7.89 ms on lock 0x7b5400124ba0. stack: 00007fa6622b5033 00007fa6622b4f30 00007fa661c13534 00000000003b4464 00000000003b43f9 00007fa668acd471 00007fa668ad740f 00007fa668ad678b 00007fa668ae1220 00007fa668ae11c8 00007fa668ae1148 00007fa668ae1103 00007fa668ae038c 00007fa66b218206 00007fa66b210808 00007fa668b1d38b
0811 18:02:42.370610 (+416868us) op_driver.cc:444] REPLICATION: finished
0811 18:02:42.411834 (+ 41224us) write_op.cc:301] APPLY: starting
0811 18:02:42.411944 (+ 110us) tablet.cc:1365] starting BulkCheckPresence
0811 18:02:42.412097 (+ 153us) tablet.cc:1368] finished BulkCheckPresence
0811 18:02:42.412143 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0811 18:02:42.420156 (+ 8013us) tablet.cc:1381] finished ApplyRowOperation cycle
0811 18:02:42.420482 (+ 326us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0811 18:02:42.420553 (+ 71us) write_op.cc:312] APPLY: finished
0811 18:02:42.420842 (+ 289us) log.cc:842] Serialized 22 byte log entry
0811 18:02:42.422094 (+ 1252us) write_op.cc:489] Releasing partition, row and schema locks
0811 18:02:42.422212 (+ 118us) write_op.cc:454] Released schema lock
0811 18:02:42.422411 (+ 199us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":26828,"apply.run_cpu_time_us":3410,"apply.run_wall_time_us":46290,"num_ops":1,"prepare.queue_time_us":50233,"prepare.run_cpu_time_us":16385,"prepare.run_wall_time_us":120299,"replication_time_us":536361,"spinlock_wait_cycles":193962112,"thread_start_us":89424,"threads_started":3,"wal-append.queue_time_us":32614}]]}
I20260811 18:02:42.470948 31087 raft_consensus.cc:1261] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Refusing update from remote peer f89e2ed2b23e4472a118ef4746b164df: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260811 18:02:42.475029 31256 consensus_queue.cc:1035] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [LEADER]: Connected to new peer: Peer: permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 18:02:42.476166 31256 consensus_queue.cc:1225] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [LEADER]: Peer 148f3fa79b0f430fa5178ae41825982c log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260811 18:02:42.480093 31084 pending_rounds.cc:76] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c: Aborting all ops after (but not including) 0
I20260811 18:02:42.480573 31084 pending_rounds.cc:97] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260811 18:02:42.480991 31084 raft_consensus.cc:2866] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260811 18:02:42.487043 31302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:02:42.487960 30959 raft_consensus.cc:1261] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Refusing update from remote peer f89e2ed2b23e4472a118ef4746b164df: Log matching property violated. Preceding OpId in replica: term: 3 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260811 18:02:42.496459 31307 consensus_queue.cc:1035] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260811 18:02:42.547215 31302 catalog_manager.cc:669] Loaded metadata for table test-table [id=975e80cdd70e47f781fed1af75d8e92f]
I20260811 18:02:42.564688 31302 tablet_loader.cc:96] loaded metadata for tablet 4414d0bbf2fc4a5488b033921fe9ff48 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:02:42.568805 31302 tablet_loader.cc:96] loaded metadata for tablet 526e9c9faff74cbb85a95853c233fdb6 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:02:42.575273 31302 tablet_loader.cc:96] loaded metadata for tablet 56c6ebe49c974aea97bdf8c98ca05ec3 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:02:42.578540 31302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:02:42.598434 31302 catalog_manager.cc:1259] Loaded cluster ID: 768dd58bfe0e482e8fc13c1742ac9359
I20260811 18:02:42.598958 31302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:02:42.605001 31302 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:02:42.613879 31302 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c: Loaded TSK: 0
I20260811 18:02:42.616760 31302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:02:42.796998 31001 heartbeater.cc:503] Master 127.27.94.252:40527 was elected leader, sending a full tablet report...
I20260811 18:02:42.807690 30880 heartbeater.cc:503] Master 127.27.94.252:40527 was elected leader, sending a full tablet report...
I20260811 18:02:42.808924 31127 heartbeater.cc:503] Master 127.27.94.252:40527 was elected leader, sending a full tablet report...
I20260811 18:02:42.823073 30601 catalog_manager.cc:5495] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 reported cstate change: term changed from 0 to 4, leader changed from <none> to f89e2ed2b23e4472a118ef4746b164df (127.27.94.193). New cstate: current_term: 4 leader_uuid: "f89e2ed2b23e4472a118ef4746b164df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } }
I20260811 18:02:42.826431 30601 catalog_manager.cc:5495] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148f3fa79b0f430fa5178ae41825982c (127.27.94.195). New cstate: current_term: 1 leader_uuid: "148f3fa79b0f430fa5178ae41825982c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } }
I20260811 18:02:43.218750 31316 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:02:43.219991 31316 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:43.288487 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:43.288502 31318 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f89e2ed2b23e4472a118ef4746b164df)
I20260811 18:02:43.289775 31318 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:43.302760 31316 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:43.308638 31316 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:43.309274 31318 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:43.309648 31316 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:43.313038 31084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:43.314220 31316 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:43.315390 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:43.320397 31319 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:02:43.321254 31319 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
W20260811 18:02:43.296591 30764 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.94.195:33295, user_credentials={real_user=slave}} blocked reactor thread for 79671.7us
I20260811 18:02:43.340065 31084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:43.346056 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
I20260811 18:02:43.345569 30764 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:43.342146 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:43.348503 31319 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:02:43.350706 30764 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:43.355113 31320 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:43.368942 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:43.371371 30903 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9; no voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:43.362751 31322 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:43.377416 31318 raft_consensus.cc:2726] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260811 18:02:43.397258 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:02:43.403999 30364 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: baad4df5e5d244749b5730df044077a4; no voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, ceb28f49354743839ed45f09a79b5b5c
I20260811 18:02:43.405896 31319 raft_consensus.cc:2726] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260811 18:02:44.032682 31320 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:44.037101 31320 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:44.058581 31348 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:44.061371 31316 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:02:44.062119 31316 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:44.066684 31316 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:44.069010 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:44.072510 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:44.077276 30762 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:44.083986 31316 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:44.059523 31348 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:44.091132 31348 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:44.101481 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:44.104242 31320 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:44.106276 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:44.106428 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:44.105690 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:44.109433 30762 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:44.110553 31320 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:44.113713 31026 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c; no voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:44.116658 31348 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:56.090330 31378 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f89e2ed2b23e4472a118ef4746b164df)
I20260811 18:02:56.091683 31378 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:56.095813 31378 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:56.234547 31389 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:56.235711 31389 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:56.240588 31389 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:56.255079 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:56.281903 31418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:02:56.282543 31418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:56.287132 31418 leader_election.cc:288] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:02:56.330171 31378 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:02:56.332957 31378 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:56.335058 31378 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:56.336941 30837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:56.340448 31424 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:56.340981 31424 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:56.342738 31413 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f89e2ed2b23e4472a118ef4746b164df)
I20260811 18:02:56.343277 31413 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:56.344009 31424 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:56.344436 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4" is_pre_election: true
I20260811 18:02:56.345067 30406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5295d3ba6a4b4a5ba4ed3e13a28771fb in term 3.
I20260811 18:02:56.346678 30470 leader_election.cc:302] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, baad4df5e5d244749b5730df044077a4; no voters:
I20260811 18:02:56.347664 31418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260811 18:02:56.347780 31413 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:56.348157 31418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 3 FOLLOWER]: Starting leader election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:02:56.348647 31418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:02:56.360983 31418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:56.363417 31418 leader_election.cc:288] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 4 election: Requested vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:02:56.365345 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4"
I20260811 18:02:56.365710 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c"
I20260811 18:02:56.366125 30406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:02:56.379415 30406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5295d3ba6a4b4a5ba4ed3e13a28771fb in term 4.
I20260811 18:02:56.379415 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:02:56.383440 30404 raft_consensus.cc:1226] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 4 FOLLOWER]: Rejecting Update request from peer ceb28f49354743839ed45f09a79b5b5c for earlier term 3. Current term is 4. Ops: []
I20260811 18:02:56.388871 30470 leader_election.cc:302] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, baad4df5e5d244749b5730df044077a4; no voters: ceb28f49354743839ed45f09a79b5b5c
I20260811 18:02:56.390280 31418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 4 FOLLOWER]: Leader election won for term 4
I20260811 18:02:56.392413 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:56.394944 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:56.396473 31086 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f89e2ed2b23e4472a118ef4746b164df in term 1.
I20260811 18:02:56.401064 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:56.401134 30764 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df; no voters: 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:56.404132 31418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 4 LEADER]: Becoming Leader. State: Replica: 5295d3ba6a4b4a5ba4ed3e13a28771fb, State: Running, Role: LEADER
I20260811 18:02:56.404974 30904 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9; no voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:56.405385 31414 raft_consensus.cc:2781] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 18:02:56.405929 31414 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting leader election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:56.407107 31378 raft_consensus.cc:2726] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260811 18:02:56.406935 31414 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:56.407440 31418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:56.419562 31084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:56.429364 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:56.431277 30904 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9; no voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:56.434844 30501 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 4 LEADER]: Rejecting Update request from peer ceb28f49354743839ed45f09a79b5b5c for earlier term 3. Current term is 4. Ops: []
I20260811 18:02:56.439564 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:56.440762 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:56.434063 30837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:56.451337 31026 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c; no voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:56.458315 31414 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:56.461387 31412 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 }, Status: INVALID_TERM, Last received: 3.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260811 18:02:56.463310 30838 raft_consensus.cc:1226] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Rejecting Update request from peer 90c3072068094afbafbc8e30ae2dc8c9 for earlier term 1. Current term is 2. Ops: []
I20260811 18:02:56.470520 31422 consensus_queue.cc:1046] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260811 18:02:56.475929 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9"
I20260811 18:02:56.484968 31414 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 election: Requested vote from peers 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:56.492396 31413 raft_consensus.cc:2726] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260811 18:02:56.510452 30404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 4 FOLLOWER]: Refusing update from remote peer 5295d3ba6a4b4a5ba4ed3e13a28771fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20260811 18:02:56.523351 31418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.018s
I20260811 18:02:56.521284 31426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5295d3ba6a4b4a5ba4ed3e13a28771fb. Latest consensus state: current_term: 4 leader_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:56.539541 31378 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:56.543787 31426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:56.591696 31431 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:02:56.593458 31431 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:56.605325 31436 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 }, Status: INVALID_TERM, Last received: 3.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20260811 18:02:56.608736 31436 consensus_queue.cc:1225] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer 5295d3ba6a4b4a5ba4ed3e13a28771fb log is divergent from this leader: its last log entry 4.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260811 18:02:56.614898 31378 raft_consensus.cc:3032] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 LEADER]: Stepping down as leader of term 1
I20260811 18:02:56.622705 31378 raft_consensus.cc:729] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Running, Role: LEADER
I20260811 18:02:56.617332 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c"
I20260811 18:02:56.625901 30620 raft_consensus.cc:3032] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 3 LEADER]: Stepping down as leader of term 3
I20260811 18:02:56.629472 31378 consensus_queue.cc:260] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
W20260811 18:02:56.635185 30577 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.94.253:34173, user_credentials={real_user=slave}} blocked reactor thread for 50072.6us
I20260811 18:02:56.648921 30501 raft_consensus.cc:1226] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 4 LEADER]: Rejecting Update request from peer ceb28f49354743839ed45f09a79b5b5c for earlier term 3. Current term is 4. Ops: []
I20260811 18:02:56.651161 31431 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:56.642262 30764 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
W20260811 18:02:56.598734 31026 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.194:38115, user_credentials={real_user=slave}} blocked reactor thread for 148782us
I20260811 18:02:56.657609 31026 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c; no voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:56.657352 31431 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260811 18:02:56.659302 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:56.627041 30620 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ceb28f49354743839ed45f09a79b5b5c, State: Running, Role: LEADER
I20260811 18:02:56.654181 31378 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:02:56.654991 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:56.664408 31432 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:56.670266 30764 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:56.675997 31389 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:56.670297 31438 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 }, Status: INVALID_TERM, Last received: 3.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.011s
I20260811 18:02:56.678791 31441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:02:56.678900 31438 consensus_queue.cc:1225] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer 5295d3ba6a4b4a5ba4ed3e13a28771fb log is divergent from this leader: its last log entry 4.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260811 18:02:56.698501 30620 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:56.702122 30620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:02:56.705753 31440 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5295d3ba6a4b4a5ba4ed3e13a28771fb. Latest consensus state: current_term: 4 leader_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:56.711908 31440 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:56.722110 30406 raft_consensus.cc:1226] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 4 FOLLOWER]: Rejecting Update request from peer ceb28f49354743839ed45f09a79b5b5c for earlier term 3. Current term is 4. Ops: []
I20260811 18:02:56.722915 31441 catalog_manager.cc:669] Loaded metadata for table test-table [id=975e80cdd70e47f781fed1af75d8e92f]
I20260811 18:02:56.732442 30620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 4 FOLLOWER]: Refusing update from remote peer 5295d3ba6a4b4a5ba4ed3e13a28771fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
W20260811 18:02:56.734850 31412 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260811 18:02:56.740302 31426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260811 18:02:56.747619 31440 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:56.750180 31440 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:56.759796 31441 tablet_loader.cc:96] loaded metadata for tablet 4414d0bbf2fc4a5488b033921fe9ff48 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:02:56.762408 31441 tablet_loader.cc:96] loaded metadata for tablet 526e9c9faff74cbb85a95853c233fdb6 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:02:56.765990 31441 tablet_loader.cc:96] loaded metadata for tablet 56c6ebe49c974aea97bdf8c98ca05ec3 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:02:56.768621 31441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:02:56.770646 31426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:56.772354 31426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:56.775704 31441 catalog_manager.cc:1259] Loaded cluster ID: 768dd58bfe0e482e8fc13c1742ac9359
I20260811 18:02:56.778641 31441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:02:56.780357 31412 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5295d3ba6a4b4a5ba4ed3e13a28771fb. Latest consensus state: current_term: 4 leader_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:56.782626 31412 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:56.784003 31438 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:56.786244 31438 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:56.792295 31441 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:02:56.798000 30743 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 18:02:56.800945 31441 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb: Loaded TSK: 0
I20260811 18:02:56.803064 31441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:02:56.814786 31378 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:02:56.815346 31378 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:56.817689 31378 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:56.818882 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:56.818882 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:56.819813 31087 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 1.
I20260811 18:02:56.819813 30838 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 2.
I20260811 18:02:56.821206 30904 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9; no voters:
I20260811 18:02:56.821916 31378 raft_consensus.cc:2781] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260811 18:02:56.822345 31378 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:02:56.822602 31378 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:56.824443 31418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:56.826390 31418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:56.832160 31378 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:56.833676 31378 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 3 election: Requested vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:56.835129 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c"
I20260811 18:02:56.835129 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df"
I20260811 18:02:56.835988 31087 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Advancing to term 3
I20260811 18:02:56.836066 30838 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Advancing to term 3
I20260811 18:02:56.846701 31087 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 3.
I20260811 18:02:56.846931 30838 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 3.
I20260811 18:02:56.848716 30903 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:02:56.849825 31378 raft_consensus.cc:2781] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 18:02:56.850383 31378 raft_consensus.cc:686] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 LEADER]: Becoming Leader. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Running, Role: LEADER
I20260811 18:02:56.852276 31378 consensus_queue.cc:237] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:56.877334 30491 catalog_manager.cc:5495] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "90c3072068094afbafbc8e30ae2dc8c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } health_report { overall_health: HEALTHY } } }
I20260811 18:02:56.920301 30879 heartbeater.cc:503] Master 127.27.94.253:34173 was elected leader, sending a full tablet report...
I20260811 18:02:56.946437 30838 raft_consensus.cc:1261] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Refusing update from remote peer 90c3072068094afbafbc8e30ae2dc8c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260811 18:02:56.948292 31378 consensus_queue.cc:1035] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260811 18:02:56.969609 30997 heartbeater.cc:503] Master 127.27.94.253:34173 was elected leader, sending a full tablet report...
I20260811 18:02:56.972465 31087 raft_consensus.cc:1261] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Refusing update from remote peer 90c3072068094afbafbc8e30ae2dc8c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260811 18:02:56.975334 31422 consensus_queue.cc:1035] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260811 18:02:56.978626 31126 heartbeater.cc:503] Master 127.27.94.253:34173 was elected leader, sending a full tablet report...
W20260811 18:02:57.437594 31122 debug-util.cc:398] Leaking SignalData structure 0x7b080020a060 after lost signal to thread 28030
W20260811 18:02:57.438791 31122 debug-util.cc:398] Leaking SignalData structure 0x7b080020a7c0 after lost signal to thread 30447
W20260811 18:02:57.439829 31122 debug-util.cc:398] Leaking SignalData structure 0x7b080020ae80 after lost signal to thread 30543
W20260811 18:02:57.440968 31122 debug-util.cc:398] Leaking SignalData structure 0x7b08001364e0 after lost signal to thread 30657
W20260811 18:02:57.442981 31122 debug-util.cc:398] Leaking SignalData structure 0x7b080009a440 after lost signal to thread 30871
W20260811 18:02:57.444134 31122 debug-util.cc:398] Leaking SignalData structure 0x7b0800227880 after lost signal to thread 30995
W20260811 18:02:57.445112 31122 debug-util.cc:398] Leaking SignalData structure 0x7b0800222820 after lost signal to thread 31125
I20260811 18:02:58.013288 31466 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5295d3ba6a4b4a5ba4ed3e13a28771fb)
I20260811 18:02:58.014184 31466 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:58.026810 31466 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:02:58.110481 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
I20260811 18:02:58.137542 31468 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5295d3ba6a4b4a5ba4ed3e13a28771fb)
I20260811 18:02:58.138651 31468 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:58.143537 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4" is_pre_election: true
I20260811 18:02:58.144345 30406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb28f49354743839ed45f09a79b5b5c in term 4.
I20260811 18:02:58.146037 31468 leader_election.cc:288] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173)
I20260811 18:02:58.146438 30583 leader_election.cc:302] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baad4df5e5d244749b5730df044077a4, ceb28f49354743839ed45f09a79b5b5c; no voters:
I20260811 18:02:58.147315 31468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260811 18:02:58.148121 31468 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5295d3ba6a4b4a5ba4ed3e13a28771fb)
I20260811 18:02:58.148840 31468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 4 FOLLOWER]: Advancing to term 5
I20260811 18:02:58.160598 31469 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f89e2ed2b23e4472a118ef4746b164df)
I20260811 18:02:58.161880 31469 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:58.163435 31468 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:58.167594 31469 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:58.168437 31468 leader_election.cc:288] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 5 election: Requested vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173)
I20260811 18:02:58.168756 30406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4"
I20260811 18:02:58.169325 30406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 4 FOLLOWER]: Advancing to term 5
I20260811 18:02:58.178364 30406 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb28f49354743839ed45f09a79b5b5c in term 5.
I20260811 18:02:58.179924 30583 leader_election.cc:302] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: baad4df5e5d244749b5730df044077a4, ceb28f49354743839ed45f09a79b5b5c; no voters:
I20260811 18:02:58.180963 31468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 5 FOLLOWER]: Leader election won for term 5
I20260811 18:02:58.186870 31468 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 5 LEADER]: Becoming Leader. State: Replica: ceb28f49354743839ed45f09a79b5b5c, State: Running, Role: LEADER
I20260811 18:02:58.188128 31468 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 4.11, Last appended by leader: 11, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:58.197934 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
I20260811 18:02:58.198734 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb28f49354743839ed45f09a79b5b5c" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb"
I20260811 18:02:58.200646 31433 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:02:58.201551 31433 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:58.205318 31433 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:58.207006 30406 raft_consensus.cc:1226] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 5 FOLLOWER]: Rejecting Update request from peer 5295d3ba6a4b4a5ba4ed3e13a28771fb for earlier term 4. Current term is 5. Ops: []
I20260811 18:02:58.208803 31464 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 }, Status: INVALID_TERM, Last received: 4.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260811 18:02:58.209535 31431 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:58.209919 31464 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 4 LEADER]: Stepping down as leader of term 4
I20260811 18:02:58.210323 31464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5295d3ba6a4b4a5ba4ed3e13a28771fb, State: Running, Role: LEADER
I20260811 18:02:58.210146 31431 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:58.211189 31464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 4.11, Last appended by leader: 11, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:02:58.212455 31464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 4 FOLLOWER]: Advancing to term 5
I20260811 18:02:58.214193 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:58.216898 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:58.217370 31424 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:02:58.217618 30956 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148f3fa79b0f430fa5178ae41825982c in term 4.
I20260811 18:02:58.217912 31424 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:02:58.218912 31026 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9; no voters:
I20260811 18:02:58.219738 31469 raft_consensus.cc:2781] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260811 18:02:58.220153 31469 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Starting leader election (detected failure of leader f89e2ed2b23e4472a118ef4746b164df)
I20260811 18:02:58.220659 31469 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Advancing to term 5
I20260811 18:02:58.220194 31424 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:58.212468 31431 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:58.224974 31431 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:02:58.224964 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:58.225904 31431 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:58.229915 31470 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb28f49354743839ed45f09a79b5b5c. Latest consensus state: current_term: 5 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:58.230760 31470 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:58.230899 31431 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:02:58.231715 31469 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:58.232465 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:02:58.235203 31469 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 election: Requested vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:02:58.236779 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9"
I20260811 18:02:58.236752 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 5 candidate_status { last_received { term: 4 index: 11 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:02:58.237815 30956 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Advancing to term 5
W20260811 18:02:58.238122 30620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46074 (request call id 7) took 171 ms (client timeout 216 ms). Trace:
W20260811 18:02:58.238538 30620 rpcz_store.cc:269] 0811 18:02:58.066707 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 18:02:58.236519 (+169812us) service_pool.cc:227] Handling call
0811 18:02:58.238031 (+ 1512us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260811 18:02:58.240777 30364 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: baad4df5e5d244749b5730df044077a4; no voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, ceb28f49354743839ed45f09a79b5b5c
I20260811 18:02:58.242174 31466 raft_consensus.cc:2726] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260811 18:02:58.250567 30956 raft_consensus.cc:2454] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148f3fa79b0f430fa5178ae41825982c in term 5.
I20260811 18:02:58.251905 31026 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9; no voters:
I20260811 18:02:58.253248 31469 raft_consensus.cc:2781] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 5 FOLLOWER]: Leader election won for term 5
I20260811 18:02:58.255750 30621 raft_consensus.cc:1226] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 5 LEADER]: Rejecting Update request from peer 5295d3ba6a4b4a5ba4ed3e13a28771fb for earlier term 4. Current term is 5. Ops: []
I20260811 18:02:58.257536 30837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:58.259507 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df"
I20260811 18:02:58.269605 31469 raft_consensus.cc:686] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 5 LEADER]: Becoming Leader. State: Replica: 148f3fa79b0f430fa5178ae41825982c, State: Running, Role: LEADER
I20260811 18:02:58.271060 31469 consensus_queue.cc:237] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:58.279773 30956 raft_consensus.cc:1226] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Rejecting Update request from peer f89e2ed2b23e4472a118ef4746b164df for earlier term 4. Current term is 5. Ops: []
I20260811 18:02:58.281180 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:58.282773 31389 consensus_queue.cc:1046] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260811 18:02:58.282964 31026 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c; no voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:58.284004 31431 raft_consensus.cc:3032] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 4 LEADER]: Stepping down as leader of term 4
I20260811 18:02:58.285183 31469 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260811 18:02:58.286080 31431 raft_consensus.cc:729] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f89e2ed2b23e4472a118ef4746b164df, State: Running, Role: LEADER
I20260811 18:02:58.287065 31431 consensus_queue.cc:260] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:02:58.288921 31431 raft_consensus.cc:3037] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 4 FOLLOWER]: Advancing to term 5
I20260811 18:02:58.292953 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:58.294450 30764 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:58.295689 31389 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:58.298719 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:58.299014 30406 raft_consensus.cc:1261] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 5 FOLLOWER]: Refusing update from remote peer ceb28f49354743839ed45f09a79b5b5c: Log matching property violated. Preceding OpId in replica: term: 4 index: 11. Preceding OpId from leader: term: 5 index: 12. (index mismatch)
I20260811 18:02:58.300700 31470 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260811 18:02:58.306191 30501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 5 FOLLOWER]: Refusing update from remote peer ceb28f49354743839ed45f09a79b5b5c: Log matching property violated. Preceding OpId in replica: term: 4 index: 11. Preceding OpId from leader: term: 5 index: 12. (index mismatch)
I20260811 18:02:58.315682 30834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:02:58.323880 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:02:58.326004 30904 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9; no voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:02:58.326233 31468 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260811 18:02:58.328107 31433 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:02:58.339102 30764 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:02:58.340528 31431 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260811 18:02:58.344602 31464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb28f49354743839ed45f09a79b5b5c. Latest consensus state: current_term: 5 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:58.349263 31464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:58.375397 31464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:58.379113 31464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:58.377059 30834 raft_consensus.cc:1261] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 5 FOLLOWER]: Refusing update from remote peer 148f3fa79b0f430fa5178ae41825982c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260811 18:02:58.383436 31469 consensus_queue.cc:1035] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
W20260811 18:02:58.393164 30904 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.195:33295, user_credentials={real_user=slave}} blocked reactor thread for 67236.4us
I20260811 18:02:58.402153 31466 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb28f49354743839ed45f09a79b5b5c. Latest consensus state: current_term: 5 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:58.404093 30956 raft_consensus.cc:1261] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Refusing update from remote peer 148f3fa79b0f430fa5178ae41825982c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260811 18:02:58.406317 31473 consensus_queue.cc:1035] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260811 18:02:58.392402 31468 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:58.408792 31468 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:58.403338 31466 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:02:58.427934 31468 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:58.429719 31468 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: LEADER
I20260811 18:02:58.453827 31466 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "ceb28f49354743839ed45f09a79b5b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:02:58.455194 31466 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 18:02:58.899997 30441 debug-util.cc:398] Leaking SignalData structure 0x7b0800244aa0 after lost signal to thread 28030
W20260811 18:02:58.901888 30441 debug-util.cc:398] Leaking SignalData structure 0x7b08002291c0 after lost signal to thread 30447
W20260811 18:02:58.903264 30441 debug-util.cc:398] Leaking SignalData structure 0x7b08002290a0 after lost signal to thread 30543
W20260811 18:02:58.904362 30441 debug-util.cc:398] Leaking SignalData structure 0x7b0800244c00 after lost signal to thread 30657
W20260811 18:02:58.905925 30441 debug-util.cc:398] Leaking SignalData structure 0x7b080023aa00 after lost signal to thread 30871
W20260811 18:02:58.907226 30441 debug-util.cc:398] Leaking SignalData structure 0x7b080014b040 after lost signal to thread 30995
W20260811 18:02:58.908438 30441 debug-util.cc:398] Leaking SignalData structure 0x7b0800256200 after lost signal to thread 31125
I20260811 18:03:03.423282 31512 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:03:03.425334 31512 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:03:03.434821 31512 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:03:03.438750 31513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:03:03.439975 31513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:03:03.440356 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
I20260811 18:03:03.443743 30501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baad4df5e5d244749b5730df044077a4 in term 5.
I20260811 18:03:03.445752 31513 leader_election.cc:288] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:03:03.447093 30362 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, baad4df5e5d244749b5730df044077a4; no voters:
I20260811 18:03:03.448654 31512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260811 18:03:03.449348 31512 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ceb28f49354743839ed45f09a79b5b5c)
I20260811 18:03:03.449985 31512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 5 FOLLOWER]: Advancing to term 6
I20260811 18:03:03.458391 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:03:03.463029 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:03:03.474525 31512 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:03:03.481017 30404 raft_consensus.cc:1226] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 6 FOLLOWER]: Rejecting Update request from peer ceb28f49354743839ed45f09a79b5b5c for earlier term 5. Current term is 6. Ops: []
I20260811 18:03:03.483711 31512 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 6 election: Requested vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:03:03.488144 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c"
I20260811 18:03:03.487435 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb"
I20260811 18:03:03.492391 30362 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: baad4df5e5d244749b5730df044077a4; no voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, ceb28f49354743839ed45f09a79b5b5c
I20260811 18:03:03.494427 31512 raft_consensus.cc:2726] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260811 18:03:03.695962 30470 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.27.94.254:42193 timed out after 0.244s (SENT) [suppressed 6 similar messages]
I20260811 18:03:03.802901 31514 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:03:03.807502 31514 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:03:03.814008 31512 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:03:03.826444 31512 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:03:03.850556 31514 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:03:03.870591 31512 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:03:03.952348 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
I20260811 18:03:03.953835 30506 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baad4df5e5d244749b5730df044077a4 in term 5.
I20260811 18:03:03.971613 30362 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, baad4df5e5d244749b5730df044077a4; no voters:
I20260811 18:03:03.974246 31512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260811 18:03:03.975024 31512 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:03:03.975744 31512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 6 FOLLOWER]: Advancing to term 7
I20260811 18:03:04.050729 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:03:04.077359 31512 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:03:04.082823 31512 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 7 election: Requested vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:03:04.083947 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c"
I20260811 18:03:04.085964 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 7 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb"
I20260811 18:03:04.089978 30362 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: baad4df5e5d244749b5730df044077a4; no voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, ceb28f49354743839ed45f09a79b5b5c
I20260811 18:03:04.092373 31512 raft_consensus.cc:2726] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260811 18:03:04.107118 30406 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57364 (request call id 22) took 652 ms (client timeout 244 ms). Trace:
W20260811 18:03:04.107827 30406 rpcz_store.cc:269] 0811 18:03:03.453803 (+ 0us) service_pool.cc:170] Inserting onto call queue
0811 18:03:04.105600 (+651797us) service_pool.cc:214] Skipping call since client already timed out
0811 18:03:04.105844 (+ 244us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260811 18:03:04.125732 31518 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:03:04.127005 31518 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:03:04.132527 31518 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
W20260811 18:03:04.200606 30470 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.013s sys 0.014s
W20260811 18:03:04.201310 30470 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.503s user 0.013s sys 0.014s
W20260811 18:03:04.201728 30470 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.94.254:42193, user_credentials={real_user=slave}} blocked reactor thread for 507167us
I20260811 18:03:04.204974 30404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" candidate_term: 6 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "baad4df5e5d244749b5730df044077a4" is_pre_election: true
I20260811 18:03:04.206869 30404 raft_consensus.cc:2354] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5295d3ba6a4b4a5ba4ed3e13a28771fb for earlier term 6. Current term is 7.
I20260811 18:03:04.209267 30470 leader_election.cc:398] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 6 pre-election: Vote denied by peer baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5295d3ba6a4b4a5ba4ed3e13a28771fb for earlier term 6. Current term is 7.
I20260811 18:03:04.209586 30470 leader_election.cc:401] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260811 18:03:04.213577 31519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 5 FOLLOWER]: Advancing to term 7
I20260811 18:03:04.242151 31519 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5295d3ba6a4b4a5ba4ed3e13a28771fb for earlier term 6. Current term is 7.
I20260811 18:03:04.256570 31520 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:03:04.257838 31520 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:03:04.270774 31521 raft_consensus.cc:482] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:03:04.272687 31521 raft_consensus.cc:504] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:03:04.279230 31521 leader_election.cc:288] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:03:04.279469 31518 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:03:04.281297 31518 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:03:04.280323 31522 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 }, Status: INVALID_TERM, Last received: 5.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.006s
I20260811 18:03:04.287854 31518 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:03:04.289462 31086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:03:04.290247 31520 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:03:04.294575 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:03:04.295598 31087 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f89e2ed2b23e4472a118ef4746b164df in term 3.
I20260811 18:03:04.300861 30764 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df; no voters:
I20260811 18:03:04.301909 31518 raft_consensus.cc:2781] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260811 18:03:04.302429 31518 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Starting leader election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:03:04.303449 31518 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:03:04.310139 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:03:04.317797 30959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:03:04.321588 30762 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:03:04.333679 30955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:03:04.334525 31518 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:03:04.337560 31518 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 4 election: Requested vote from peers 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:03:04.338398 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c"
I20260811 18:03:04.340003 31087 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:03:04.341970 30404 raft_consensus.cc:1226] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Rejecting Update request from peer ceb28f49354743839ed45f09a79b5b5c for earlier term 5. Current term is 7. Ops: []
I20260811 18:03:04.342813 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9"
I20260811 18:03:04.344695 31523 consensus_queue.cc:1046] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 }, Status: INVALID_TERM, Last received: 5.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.002s
I20260811 18:03:04.347743 30837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:03:04.349581 30836 raft_consensus.cc:1226] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 4 FOLLOWER]: Rejecting Update request from peer 90c3072068094afbafbc8e30ae2dc8c9 for earlier term 3. Current term is 4. Ops: []
I20260811 18:03:04.358866 30834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:03:04.358152 31505 consensus_queue.cc:1046] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260811 18:03:04.360818 31505 raft_consensus.cc:3032] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 LEADER]: Stepping down as leader of term 3
I20260811 18:03:04.361198 31505 raft_consensus.cc:729] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Running, Role: LEADER
I20260811 18:03:04.362310 31505 consensus_queue.cc:260] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:03:04.366433 31084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526e9c9faff74cbb85a95853c233fdb6" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:03:04.366923 31085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:03:04.367652 31526 raft_consensus.cc:2726] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260811 18:03:04.363469 31522 raft_consensus.cc:3032] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 5 LEADER]: Stepping down as leader of term 5
I20260811 18:03:04.368131 31505 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 3 FOLLOWER]: Advancing to term 4
I20260811 18:03:04.368474 31522 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ceb28f49354743839ed45f09a79b5b5c, State: Running, Role: LEADER
I20260811 18:03:04.370679 30904 leader_election.cc:302] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9; no voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:03:04.372236 31520 raft_consensus.cc:2726] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260811 18:03:04.372093 30836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "148f3fa79b0f430fa5178ae41825982c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df" is_pre_election: true
I20260811 18:03:04.373077 31512 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:03:04.373544 30836 raft_consensus.cc:2379] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148f3fa79b0f430fa5178ae41825982c in current term 4: Already voted for candidate f89e2ed2b23e4472a118ef4746b164df in this term.
I20260811 18:03:04.373597 31512 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:03:04.338798 31518 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:03:04.375533 31518 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:03:04.370397 31522 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 5.12, Last appended by leader: 12, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:03:04.376248 31512 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:03:04.376526 31026 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c; no voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:03:04.378125 30837 raft_consensus.cc:2454] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 1.
I20260811 18:03:04.379360 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" is_pre_election: true
I20260811 18:03:04.380324 30501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baad4df5e5d244749b5730df044077a4 in term 7.
I20260811 18:03:04.379823 31522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 5 FOLLOWER]: Advancing to term 6
I20260811 18:03:04.383309 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c" is_pre_election: true
I20260811 18:03:04.384624 30903 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c
I20260811 18:03:04.389091 31520 raft_consensus.cc:2781] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 18:03:04.388628 30362 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, baad4df5e5d244749b5730df044077a4; no voters:
I20260811 18:03:04.383846 31087 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f89e2ed2b23e4472a118ef4746b164df in term 4.
I20260811 18:03:04.389873 31085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:03:04.390380 31512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260811 18:03:04.389623 31520 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 148f3fa79b0f430fa5178ae41825982c)
I20260811 18:03:04.391314 31520 raft_consensus.cc:3037] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:03:04.391570 31510 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260811 18:03:04.394589 31086 raft_consensus.cc:1226] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Rejecting Update request from peer 90c3072068094afbafbc8e30ae2dc8c9 for earlier term 3. Current term is 4. Ops: []
I20260811 18:03:04.398437 31512 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:03:04.399379 31512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 7 FOLLOWER]: Advancing to term 8
I20260811 18:03:04.387451 31518 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:03:04.409610 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:03:04.418352 30764 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, f89e2ed2b23e4472a118ef4746b164df; no voters: 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:03:04.421080 31518 raft_consensus.cc:2781] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 4 FOLLOWER]: Leader election won for term 4
I20260811 18:03:04.421844 31518 raft_consensus.cc:686] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 4 LEADER]: Becoming Leader. State: Replica: f89e2ed2b23e4472a118ef4746b164df, State: Running, Role: LEADER
I20260811 18:03:04.420989 31512 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:03:04.423638 31518 consensus_queue.cc:237] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:03:04.426177 30501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb"
I20260811 18:03:04.426952 30501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 7 FOLLOWER]: Advancing to term 8
I20260811 18:03:04.417325 30762 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:03:04.430035 31524 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:03:04.440598 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "baad4df5e5d244749b5730df044077a4" candidate_term: 8 candidate_status { last_received { term: 5 index: 12 } } ignore_live_leader: false dest_uuid: "ceb28f49354743839ed45f09a79b5b5c"
I20260811 18:03:04.445422 30621 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baad4df5e5d244749b5730df044077a4 in term 6.
I20260811 18:03:04.446256 31512 leader_election.cc:288] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 8 election: Requested vote from peers 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173), ceb28f49354743839ed45f09a79b5b5c (127.27.94.252:40527)
I20260811 18:03:04.445962 31520 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:03:04.448329 30956 raft_consensus.cc:1226] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Rejecting Update request from peer 148f3fa79b0f430fa5178ae41825982c for earlier term 1. Current term is 2. Ops: []
I20260811 18:03:04.453034 30838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f89e2ed2b23e4472a118ef4746b164df"
I20260811 18:03:04.455237 30501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baad4df5e5d244749b5730df044077a4 in term 8.
I20260811 18:03:04.455338 31527 consensus_queue.cc:1046] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260811 18:03:04.456841 30362 leader_election.cc:302] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, baad4df5e5d244749b5730df044077a4; no voters: ceb28f49354743839ed45f09a79b5b5c
I20260811 18:03:04.458225 31527 raft_consensus.cc:3032] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 1 LEADER]: Stepping down as leader of term 1
I20260811 18:03:04.459898 31512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 8 FOLLOWER]: Leader election won for term 8
I20260811 18:03:04.460631 31527 raft_consensus.cc:729] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 148f3fa79b0f430fa5178ae41825982c, State: Running, Role: LEADER
I20260811 18:03:04.462586 31527 consensus_queue.cc:260] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:03:04.465261 31527 raft_consensus.cc:3037] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:03:04.468904 31520 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 2 election: Requested vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:03:04.471468 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c"
I20260811 18:03:04.473515 31512 raft_consensus.cc:686] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 8 LEADER]: Becoming Leader. State: Replica: baad4df5e5d244749b5730df044077a4, State: Running, Role: LEADER
I20260811 18:03:04.475190 31512 consensus_queue.cc:237] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 5.12, Last appended by leader: 12, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:03:04.487346 31531 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baad4df5e5d244749b5730df044077a4. Latest consensus state: current_term: 8 leader_uuid: "baad4df5e5d244749b5730df044077a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:03:04.488869 31531 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:04.491981 31087 raft_consensus.cc:2454] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 2.
I20260811 18:03:04.492911 31534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:03:04.494953 30904 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9; no voters: f89e2ed2b23e4472a118ef4746b164df
I20260811 18:03:04.496104 31520 raft_consensus.cc:2781] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 18:03:04.496994 31520 raft_consensus.cc:686] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 LEADER]: Becoming Leader. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Running, Role: LEADER
I20260811 18:03:04.499301 31520 consensus_queue.cc:237] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:03:04.511276 31534 catalog_manager.cc:669] Loaded metadata for table test-table [id=975e80cdd70e47f781fed1af75d8e92f]
I20260811 18:03:04.521905 31087 raft_consensus.cc:1261] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Refusing update from remote peer f89e2ed2b23e4472a118ef4746b164df: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260811 18:03:04.524773 31534 tablet_loader.cc:96] loaded metadata for tablet 4414d0bbf2fc4a5488b033921fe9ff48 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:03:04.524369 31518 consensus_queue.cc:1035] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [LEADER]: Connected to new peer: Peer: permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260811 18:03:04.527197 31534 tablet_loader.cc:96] loaded metadata for tablet 526e9c9faff74cbb85a95853c233fdb6 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:03:04.529578 31534 tablet_loader.cc:96] loaded metadata for tablet 56c6ebe49c974aea97bdf8c98ca05ec3 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:03:04.531514 31534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:03:04.532562 30956 raft_consensus.cc:1261] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Refusing update from remote peer f89e2ed2b23e4472a118ef4746b164df: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260811 18:03:04.534417 31518 consensus_queue.cc:1035] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260811 18:03:04.538319 31534 catalog_manager.cc:1259] Loaded cluster ID: 768dd58bfe0e482e8fc13c1742ac9359
I20260811 18:03:04.538828 31534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:03:04.550650 31534 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:03:04.563509 31534 catalog_manager.cc:5879] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4: Loaded TSK: 0
I20260811 18:03:04.566725 31534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:03:04.570358 30621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 6 FOLLOWER]: Advancing to term 8
I20260811 18:03:04.593062 30501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 8 FOLLOWER]: Refusing update from remote peer baad4df5e5d244749b5730df044077a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 12. Preceding OpId from leader: term: 8 index: 13. (index mismatch)
I20260811 18:03:04.596841 31531 consensus_queue.cc:1035] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260811 18:03:04.603233 30621 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 8 FOLLOWER]: Refusing update from remote peer baad4df5e5d244749b5730df044077a4: Log matching property violated. Preceding OpId in replica: term: 5 index: 12. Preceding OpId from leader: term: 8 index: 13. (index mismatch)
I20260811 18:03:04.603935 30838 raft_consensus.cc:3037] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:03:04.613756 31087 raft_consensus.cc:1261] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 2 FOLLOWER]: Refusing update from remote peer 90c3072068094afbafbc8e30ae2dc8c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260811 18:03:04.627153 31520 consensus_queue.cc:1035] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260811 18:03:04.627794 30838 raft_consensus.cc:1261] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Refusing update from remote peer 90c3072068094afbafbc8e30ae2dc8c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260811 18:03:04.631021 31505 consensus_queue.cc:1035] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260811 18:03:04.634442 31519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader baad4df5e5d244749b5730df044077a4. Latest consensus state: current_term: 8 leader_uuid: "baad4df5e5d244749b5730df044077a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:03:04.635677 31519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:03:04.636065 31524 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:03:04.644227 31524 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:03:04.641516 30883 heartbeater.cc:503] Master 127.27.94.254:42193 was elected leader, sending a full tablet report...
I20260811 18:03:04.648895 31531 consensus_queue.cc:1035] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260811 18:03:04.650388 31524 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:03:04.651755 31085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:03:04.660787 30386 catalog_manager.cc:5495] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 reported cstate change: term changed from 3 to 4, leader changed from 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194) to f89e2ed2b23e4472a118ef4746b164df (127.27.94.193). New cstate: current_term: 4 leader_uuid: "f89e2ed2b23e4472a118ef4746b164df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } }
I20260811 18:03:04.664536 30955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:03:04.674582 30385 catalog_manager.cc:5495] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c reported cstate change: term changed from 1 to 2, leader changed from 148f3fa79b0f430fa5178ae41825982c (127.27.94.195) to 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194). New cstate: current_term: 2 leader_uuid: "90c3072068094afbafbc8e30ae2dc8c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } }
I20260811 18:03:04.692133 30762 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:03:04.711184 31512 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "baad4df5e5d244749b5730df044077a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:03:04.712419 31512 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:04.731647 31547 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260811 18:03:04.735137 31519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "baad4df5e5d244749b5730df044077a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:03:04.739784 31519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:03:04.745970 31522 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader baad4df5e5d244749b5730df044077a4. Latest consensus state: current_term: 8 leader_uuid: "baad4df5e5d244749b5730df044077a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:03:04.747224 31522 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:03:04.770334 31531 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "baad4df5e5d244749b5730df044077a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:03:04.771457 31531 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:04.826916 31128 heartbeater.cc:503] Master 127.27.94.254:42193 was elected leader, sending a full tablet report...
I20260811 18:03:04.827608 31522 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "baad4df5e5d244749b5730df044077a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:03:04.828892 31522 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 18:03:04.848505 31002 heartbeater.cc:503] Master 127.27.94.254:42193 was elected leader, sending a full tablet report...
I20260811 18:03:04.851650 30384 catalog_manager.cc:5495] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df reported cstate change: term changed from 4 to 5, leader changed from f89e2ed2b23e4472a118ef4746b164df (127.27.94.193) to 148f3fa79b0f430fa5178ae41825982c (127.27.94.195). New cstate: current_term: 5 leader_uuid: "148f3fa79b0f430fa5178ae41825982c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } }
W20260811 18:03:04.869035 30650 debug-util.cc:398] Leaking SignalData structure 0x7b08000bc820 after lost signal to thread 28030
W20260811 18:03:04.870275 30650 debug-util.cc:398] Leaking SignalData structure 0x7b08000c84c0 after lost signal to thread 30447
W20260811 18:03:04.871281 30650 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8b80 after lost signal to thread 30543
W20260811 18:03:04.872339 30650 debug-util.cc:398] Leaking SignalData structure 0x7b0800118b40 after lost signal to thread 30657
W20260811 18:03:04.873749 30650 debug-util.cc:398] Leaking SignalData structure 0x7b080023efa0 after lost signal to thread 30871
W20260811 18:03:04.875133 30650 debug-util.cc:398] Leaking SignalData structure 0x7b08000f5800 after lost signal to thread 30995
W20260811 18:03:04.880067 30650 debug-util.cc:398] Leaking SignalData structure 0x7b0800120e20 after lost signal to thread 31125
W20260811 18:03:04.880676 30650 debug-util.cc:398] Leaking SignalData structure 0x7b0800120d00 after lost signal to thread 31471
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:783: Failure
Expected equality of these values:
live_row_count
Which is: 67
table_info->GetMetrics()->live_row_count->value()
Which is: 0
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:767: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:785: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 782, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:822: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260811 18:03:14.833076 31471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:03:14.847059 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
W20260811 18:03:14.863638 30903 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260811 18:03:14.866190 31471 catalog_manager.cc:669] Loaded metadata for table test-table [id=975e80cdd70e47f781fed1af75d8e92f]
W20260811 18:03:14.889791 30903 consensus_peers.cc:487] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 -> Peer f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821): Couldn't send request to peer f89e2ed2b23e4472a118ef4746b164df. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260811 18:03:14.893004 31471 tablet_loader.cc:96] loaded metadata for tablet 4414d0bbf2fc4a5488b033921fe9ff48 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:03:14.894757 31471 tablet_loader.cc:96] loaded metadata for tablet 526e9c9faff74cbb85a95853c233fdb6 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:03:14.896000 31471 tablet_loader.cc:96] loaded metadata for tablet 56c6ebe49c974aea97bdf8c98ca05ec3 (table test-table [id=975e80cdd70e47f781fed1af75d8e92f])
I20260811 18:03:14.897536 31471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:03:14.907109 31471 catalog_manager.cc:1259] Loaded cluster ID: 768dd58bfe0e482e8fc13c1742ac9359
I20260811 18:03:14.907614 31471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260811 18:03:14.919643 31026 consensus_peers.cc:487] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c -> Peer f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821): Couldn't send request to peer f89e2ed2b23e4472a118ef4746b164df. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260811 18:03:14.923732 31471 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:03:14.932633 31588 raft_consensus.cc:482] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 90c3072068094afbafbc8e30ae2dc8c9)
I20260811 18:03:14.934223 31588 raft_consensus.cc:504] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } }
I20260811 18:03:14.938011 31471 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c: Loaded TSK: 0
I20260811 18:03:14.939178 30956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "90c3072068094afbafbc8e30ae2dc8c9" is_pre_election: true
I20260811 18:03:14.939165 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4414d0bbf2fc4a5488b033921fe9ff48" candidate_uuid: "f89e2ed2b23e4472a118ef4746b164df" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:03:14.940626 31588 leader_election.cc:288] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:03:14.940910 31471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:03:14.941828 30764 leader_election.cc:302] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f89e2ed2b23e4472a118ef4746b164df; no voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9
I20260811 18:03:14.943120 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:14.943234 31588 raft_consensus.cc:2726] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260811 18:03:14.944401 28027 tablet_replica.cc:330] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df: stopping tablet replica
I20260811 18:03:14.945451 28027 raft_consensus.cc:2229] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 5 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:14.946895 28027 raft_consensus.cc:2258] T 526e9c9faff74cbb85a95853c233fdb6 P f89e2ed2b23e4472a118ef4746b164df [term 5 FOLLOWER]: Raft consensus is shut down!
W20260811 18:03:14.949862 31471 catalog_manager.cc:1409] Failed waiting for node to catch up after master election: Illegal state: Node ceb28f49354743839ed45f09a79b5b5c not leader. Raft Consensus state: current_term: 8 leader_uuid: "baad4df5e5d244749b5730df044077a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
W20260811 18:03:14.952389 31471 catalog_manager.cc:1409] Failed waiting for node to catch up after master election: Illegal state: Node ceb28f49354743839ed45f09a79b5b5c not leader. Raft Consensus state: current_term: 8 leader_uuid: "baad4df5e5d244749b5730df044077a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } } }
I20260811 18:03:14.954797 28027 tablet_replica.cc:330] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df: stopping tablet replica
I20260811 18:03:14.956157 28027 raft_consensus.cc:2229] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:14.957572 28027 raft_consensus.cc:2258] T 4414d0bbf2fc4a5488b033921fe9ff48 P f89e2ed2b23e4472a118ef4746b164df [term 2 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:14.960263 28027 tablet_replica.cc:330] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df: stopping tablet replica
I20260811 18:03:14.960853 28027 raft_consensus.cc:2229] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 4 LEADER]: Raft consensus shutting down.
I20260811 18:03:14.961843 28027 raft_consensus.cc:2258] T 56c6ebe49c974aea97bdf8c98ca05ec3 P f89e2ed2b23e4472a118ef4746b164df [term 4 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:14.994098 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:03:15.028602 28027 tablet_server.cc:178] TabletServer@127.27.94.194:0 shutting down...
W20260811 18:03:15.061093 31026 consensus_peers.cc:487] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c -> Peer 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115): Couldn't send request to peer 90c3072068094afbafbc8e30ae2dc8c9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260811 18:03:15.080328 31606 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f89e2ed2b23e4472a118ef4746b164df)
I20260811 18:03:15.080926 31606 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:03:15.083438 31606 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:03:15.084363 31609 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f89e2ed2b23e4472a118ef4746b164df)
I20260811 18:03:15.085119 31609 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:03:15.088232 31609 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115)
I20260811 18:03:15.093901 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c" is_pre_election: true
I20260811 18:03:15.094726 31087 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 4.
I20260811 18:03:15.096069 30904 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9; no voters:
W20260811 18:03:15.097618 31026 leader_election.cc:334] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821): Network error: Client connection negotiation failed: client connection to 127.27.94.193:38821: connect: Connection refused (error 111)
I20260811 18:03:15.097698 31606 raft_consensus.cc:2781] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260811 18:03:15.098677 31606 raft_consensus.cc:482] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f89e2ed2b23e4472a118ef4746b164df)
W20260811 18:03:15.098959 31026 leader_election.cc:334] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 90c3072068094afbafbc8e30ae2dc8c9 (127.27.94.194:38115): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260811 18:03:15.099175 31606 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 4 FOLLOWER]: Advancing to term 5
I20260811 18:03:15.099486 31026 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c; no voters: 90c3072068094afbafbc8e30ae2dc8c9, f89e2ed2b23e4472a118ef4746b164df
I20260811 18:03:15.100570 31609 raft_consensus.cc:2726] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260811 18:03:15.100786 30903 leader_election.cc:334] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821): Network error: Client connection negotiation failed: client connection to 127.27.94.193:38821: connect: Connection refused (error 111)
I20260811 18:03:15.109126 31606 raft_consensus.cc:504] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:03:15.113454 31606 leader_election.cc:288] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 5 election: Requested vote from peers f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821), 148f3fa79b0f430fa5178ae41825982c (127.27.94.195:33295)
I20260811 18:03:15.114988 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:15.114871 31087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56c6ebe49c974aea97bdf8c98ca05ec3" candidate_uuid: "90c3072068094afbafbc8e30ae2dc8c9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "148f3fa79b0f430fa5178ae41825982c"
I20260811 18:03:15.116288 31087 raft_consensus.cc:3037] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 4 FOLLOWER]: Advancing to term 5
I20260811 18:03:15.116850 28027 tablet_replica.cc:330] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9: stopping tablet replica
I20260811 18:03:15.118026 28027 raft_consensus.cc:2229] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 LEADER]: Raft consensus shutting down.
I20260811 18:03:15.119251 28027 raft_consensus.cc:2258] T 4414d0bbf2fc4a5488b033921fe9ff48 P 90c3072068094afbafbc8e30ae2dc8c9 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260811 18:03:15.119717 30903 leader_election.cc:334] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f89e2ed2b23e4472a118ef4746b164df (127.27.94.193:38821): Network error: Client connection negotiation failed: client connection to 127.27.94.193:38821: connect: Connection refused (error 111)
I20260811 18:03:15.121964 28027 tablet_replica.cc:330] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9: stopping tablet replica
I20260811 18:03:15.122694 28027 raft_consensus.cc:2229] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:15.123351 28027 raft_consensus.cc:2258] T 526e9c9faff74cbb85a95853c233fdb6 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:15.123783 31087 raft_consensus.cc:2454] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c3072068094afbafbc8e30ae2dc8c9 in term 5.
I20260811 18:03:15.125666 30904 leader_election.cc:302] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148f3fa79b0f430fa5178ae41825982c, 90c3072068094afbafbc8e30ae2dc8c9; no voters: f89e2ed2b23e4472a118ef4746b164df
I20260811 18:03:15.126824 28027 tablet_replica.cc:330] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9: stopping tablet replica
I20260811 18:03:15.127704 31606 raft_consensus.cc:2781] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Leader election won for term 5
I20260811 18:03:15.132521 31606 raft_consensus.cc:686] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 LEADER]: Becoming Leader. State: Replica: 90c3072068094afbafbc8e30ae2dc8c9, State: Running, Role: LEADER
I20260811 18:03:15.137614 31606 consensus_queue.cc:237] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f89e2ed2b23e4472a118ef4746b164df" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 38821 } } peers { permanent_uuid: "148f3fa79b0f430fa5178ae41825982c" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33295 } } peers { permanent_uuid: "90c3072068094afbafbc8e30ae2dc8c9" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 38115 } }
I20260811 18:03:15.142061 28027 raft_consensus.cc:2229] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 LEADER]: Raft consensus shutting down.
I20260811 18:03:15.144907 28027 pending_rounds.cc:62] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9: Trying to abort 1 pending ops.
I20260811 18:03:15.145337 28027 pending_rounds.cc:68] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9: Aborting op as it isn't in flight: id { term: 5 index: 5 } timestamp: 7317386834496716800 op_type: NO_OP noop_request { }
I20260811 18:03:15.146019 28027 raft_consensus.cc:2866] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260811 18:03:15.146418 28027 raft_consensus.cc:2258] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 90c3072068094afbafbc8e30ae2dc8c9 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:15.182236 28027 tablet_server.cc:194] TabletServer@127.27.94.194:0 shutdown complete.
I20260811 18:03:15.205603 28027 tablet_server.cc:178] TabletServer@127.27.94.195:0 shutting down...
I20260811 18:03:15.270267 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:15.271417 28027 tablet_replica.cc:330] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c: stopping tablet replica
I20260811 18:03:15.272783 28027 raft_consensus.cc:2229] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 5 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:15.273548 28027 raft_consensus.cc:2258] T 56c6ebe49c974aea97bdf8c98ca05ec3 P 148f3fa79b0f430fa5178ae41825982c [term 5 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:15.275876 28027 tablet_replica.cc:330] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c: stopping tablet replica
I20260811 18:03:15.276454 28027 raft_consensus.cc:2229] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 5 LEADER]: Raft consensus shutting down.
I20260811 18:03:15.277211 28027 raft_consensus.cc:2258] T 526e9c9faff74cbb85a95853c233fdb6 P 148f3fa79b0f430fa5178ae41825982c [term 5 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:15.279153 28027 tablet_replica.cc:330] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c: stopping tablet replica
I20260811 18:03:15.279709 28027 raft_consensus.cc:2229] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:15.280303 28027 raft_consensus.cc:2258] T 4414d0bbf2fc4a5488b033921fe9ff48 P 148f3fa79b0f430fa5178ae41825982c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:15.349506 28027 tablet_server.cc:194] TabletServer@127.27.94.195:0 shutdown complete.
I20260811 18:03:15.374773 28027 master.cc:537] Master@127.27.94.254:42193 shutting down...
I20260811 18:03:15.420691 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 8 LEADER]: Raft consensus shutting down.
I20260811 18:03:15.422333 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4 [term 8 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:15.422783 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P baad4df5e5d244749b5730df044077a4: stopping tablet replica
I20260811 18:03:15.495314 28027 master.cc:552] Master@127.27.94.254:42193 shutdown complete.
I20260811 18:03:15.521806 28027 master.cc:537] Master@127.27.94.253:34173 shutting down...
I20260811 18:03:15.564990 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 8 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:15.565802 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb [term 8 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:15.566516 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5295d3ba6a4b4a5ba4ed3e13a28771fb: stopping tablet replica
I20260811 18:03:15.567452 31615 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader baad4df5e5d244749b5730df044077a4)
I20260811 18:03:15.568145 31615 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baad4df5e5d244749b5730df044077a4" member_type: VOTER last_known_addr { host: "127.27.94.254" port: 42193 } } peers { permanent_uuid: "5295d3ba6a4b4a5ba4ed3e13a28771fb" member_type: VOTER last_known_addr { host: "127.27.94.253" port: 34173 } } peers { permanent_uuid: "ceb28f49354743839ed45f09a79b5b5c" member_type: VOTER last_known_addr { host: "127.27.94.252" port: 40527 } }
I20260811 18:03:15.570995 31615 leader_election.cc:288] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193), 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173)
W20260811 18:03:15.610956 30577 connection.cc:535] client connection to 127.27.94.253:34173 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 18:03:15.611517 30577 leader_election.cc:334] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5295d3ba6a4b4a5ba4ed3e13a28771fb (127.27.94.253:34173): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 18:03:15.612627 30583 leader_election.cc:334] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer baad4df5e5d244749b5730df044077a4 (127.27.94.254:42193): Network error: Client connection negotiation failed: client connection to 127.27.94.254:42193: connect: Connection refused (error 111)
I20260811 18:03:15.612996 30583 leader_election.cc:302] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ceb28f49354743839ed45f09a79b5b5c; no voters: 5295d3ba6a4b4a5ba4ed3e13a28771fb, baad4df5e5d244749b5730df044077a4
I20260811 18:03:15.613704 31615 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260811 18:03:15.633369 28027 master.cc:552] Master@127.27.94.253:34173 shutdown complete.
I20260811 18:03:15.662170 28027 master.cc:537] Master@127.27.94.252:40527 shutting down...
I20260811 18:03:15.701722 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 8 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:15.702371 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c [term 8 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:15.702739 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P ceb28f49354743839ed45f09a79b5b5c: stopping tablet replica
I20260811 18:03:15.765408 28027 master.cc:552] Master@127.27.94.252:40527 shutdown complete.
I20260811 18:03:15.808521 28027 test_util.cc:175] -----------------------------------------------
I20260811 18:03:15.808771 28027 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1786471338663725-28027-0
[ FAILED ] TsTabletManagerITest.TestTableStats (40419 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20260811 18:03:15.818548 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:36401
I20260811 18:03:15.820212 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:15.828351 31626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:15.828701 31627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:15.832531 31630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:15.834199 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:15.837662 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:15.838032 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:15.838172 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471395838161 us; error 0 us; skew 500 ppm
I20260811 18:03:15.838778 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:15.843600 28027 webserver.cc:460] Webserver started at http://127.27.94.254:43787/ using document root <none> and password file <none>
I20260811 18:03:15.844213 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:15.844394 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:15.844641 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:15.845858 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "3aa40f20a00a44ddbbfc0de40118c4df"
format_stamp: "Formatted at 2026-08-11 18:03:15 on dist-test-slave-kvfs"
I20260811 18:03:15.850873 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 18:03:15.855176 31638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:15.856562 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 18:03:15.857005 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "3aa40f20a00a44ddbbfc0de40118c4df"
format_stamp: "Formatted at 2026-08-11 18:03:15 on dist-test-slave-kvfs"
I20260811 18:03:15.857590 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:15.871312 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:15.872792 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:15.931061 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:36401
I20260811 18:03:15.931224 31725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:36401 every 8 connection(s)
I20260811 18:03:15.935956 31726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:15.950030 31726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df: Bootstrap starting.
I20260811 18:03:15.955443 31726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:15.961395 31726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df: No bootstrap required, opened a new log
I20260811 18:03:15.964185 31726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40f20a00a44ddbbfc0de40118c4df" member_type: VOTER }
I20260811 18:03:15.964682 31726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:15.964861 31726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa40f20a00a44ddbbfc0de40118c4df, State: Initialized, Role: FOLLOWER
I20260811 18:03:15.965476 31726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40f20a00a44ddbbfc0de40118c4df" member_type: VOTER }
I20260811 18:03:15.965905 31726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:15.966089 31726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:15.966328 31726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:15.971997 31726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40f20a00a44ddbbfc0de40118c4df" member_type: VOTER }
I20260811 18:03:15.972607 31726 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aa40f20a00a44ddbbfc0de40118c4df; no voters:
I20260811 18:03:15.973891 31726 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:15.974298 31733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:15.975894 31733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 1 LEADER]: Becoming Leader. State: Replica: 3aa40f20a00a44ddbbfc0de40118c4df, State: Running, Role: LEADER
I20260811 18:03:15.976636 31733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40f20a00a44ddbbfc0de40118c4df" member_type: VOTER }
I20260811 18:03:15.977532 31726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:03:15.982192 31734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aa40f20a00a44ddbbfc0de40118c4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40f20a00a44ddbbfc0de40118c4df" member_type: VOTER } }
I20260811 18:03:15.982007 31735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa40f20a00a44ddbbfc0de40118c4df. Latest consensus state: current_term: 1 leader_uuid: "3aa40f20a00a44ddbbfc0de40118c4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aa40f20a00a44ddbbfc0de40118c4df" member_type: VOTER } }
I20260811 18:03:15.983044 31734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:15.983283 31735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:15.985405 31743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:03:15.991058 31743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:03:15.995795 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:03:16.003773 31743 catalog_manager.cc:1347] Generated new cluster ID: 5e09b203897c43899be85d43ccd8940d
I20260811 18:03:16.004133 31743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:03:16.027266 31743 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:03:16.029309 31743 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:03:16.054979 31743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df: Generated new TSK 0
I20260811 18:03:16.055927 31743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:03:16.068692 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:16.077598 31763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:16.078251 31765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:16.081085 31768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:16.082316 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:16.083645 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:16.083847 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:16.083964 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471396083956 us; error 0 us; skew 500 ppm
I20260811 18:03:16.084642 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:16.087651 28027 webserver.cc:460] Webserver started at http://127.27.94.193:44731/ using document root <none> and password file <none>
I20260811 18:03:16.088362 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:16.088562 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:16.088857 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:16.090108 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "6bc75d69e2524f61b0d645f74311b9a1"
format_stamp: "Formatted at 2026-08-11 18:03:16 on dist-test-slave-kvfs"
I20260811 18:03:16.095819 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260811 18:03:16.102250 31775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:16.103206 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260811 18:03:16.103556 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "6bc75d69e2524f61b0d645f74311b9a1"
format_stamp: "Formatted at 2026-08-11 18:03:16 on dist-test-slave-kvfs"
I20260811 18:03:16.103879 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:16.114813 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:16.116580 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:16.118220 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:03:16.120828 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:16.121062 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:16.121317 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:16.121495 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:16.174978 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:41455
I20260811 18:03:16.175269 31872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:41455 every 8 connection(s)
I20260811 18:03:16.180459 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:16.190737 31877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:16.192361 31878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:16.196708 31883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:16.199157 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:16.200577 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:16.200791 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:16.201006 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471396200975 us; error 0 us; skew 500 ppm
I20260811 18:03:16.201526 31873 heartbeater.cc:347] Connected to a master server at 127.27.94.254:36401
I20260811 18:03:16.201642 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:16.201992 31873 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:16.202868 31873 heartbeater.cc:511] Master 127.27.94.254:36401 requested a full tablet report, sending...
I20260811 18:03:16.204869 28027 webserver.cc:460] Webserver started at http://127.27.94.194:34319/ using document root <none> and password file <none>
I20260811 18:03:16.205457 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:16.205677 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:16.206053 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:16.206415 31668 ts_manager.cc:194] Registered new tserver with Master: 6bc75d69e2524f61b0d645f74311b9a1 (127.27.94.193:41455)
I20260811 18:03:16.207412 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-1-root/instance:
uuid: "1cb7900fae9e44d9bffc6a9d8537996f"
format_stamp: "Formatted at 2026-08-11 18:03:16 on dist-test-slave-kvfs"
I20260811 18:03:16.209085 31668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34442
I20260811 18:03:16.216642 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20260811 18:03:16.221588 31891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:16.223366 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260811 18:03:16.223784 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-1-root
uuid: "1cb7900fae9e44d9bffc6a9d8537996f"
format_stamp: "Formatted at 2026-08-11 18:03:16 on dist-test-slave-kvfs"
I20260811 18:03:16.224515 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:16.245163 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:16.247202 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:16.249296 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:03:16.251891 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:16.252177 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260811 18:03:16.252460 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:16.252636 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:16.309811 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.194:42301
I20260811 18:03:16.309937 31985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.194:42301 every 8 connection(s)
I20260811 18:03:16.314841 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:16.325235 31995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:16.326686 31999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:16.329885 28027 server_base.cc:1034] running on GCE node
W20260811 18:03:16.330169 32006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:16.331387 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:16.331681 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:16.331913 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471396331891 us; error 0 us; skew 500 ppm
I20260811 18:03:16.332593 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:16.335439 31989 heartbeater.cc:347] Connected to a master server at 127.27.94.254:36401
I20260811 18:03:16.335973 31989 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:16.335970 28027 webserver.cc:460] Webserver started at http://127.27.94.195:41589/ using document root <none> and password file <none>
I20260811 18:03:16.336838 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:16.337059 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:16.337090 31989 heartbeater.cc:511] Master 127.27.94.254:36401 requested a full tablet report, sending...
I20260811 18:03:16.337348 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:16.338888 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-2-root/instance:
uuid: "9fe56ae4a1fb421f9217cdfeaf56c635"
format_stamp: "Formatted at 2026-08-11 18:03:16 on dist-test-slave-kvfs"
I20260811 18:03:16.339545 31668 ts_manager.cc:194] Registered new tserver with Master: 1cb7900fae9e44d9bffc6a9d8537996f (127.27.94.194:42301)
I20260811 18:03:16.341359 31668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34444
I20260811 18:03:16.344127 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260811 18:03:16.347990 32012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:16.349035 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 18:03:16.349377 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-2-root
uuid: "9fe56ae4a1fb421f9217cdfeaf56c635"
format_stamp: "Formatted at 2026-08-11 18:03:16 on dist-test-slave-kvfs"
I20260811 18:03:16.349711 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:16.382768 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:16.384755 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:16.386824 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:03:16.390506 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:16.390874 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:16.391186 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:16.391372 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:16.443056 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.195:33537
I20260811 18:03:16.443291 32102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.195:33537 every 8 connection(s)
I20260811 18:03:16.462297 32103 heartbeater.cc:347] Connected to a master server at 127.27.94.254:36401
I20260811 18:03:16.462803 32103 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:16.463897 32103 heartbeater.cc:511] Master 127.27.94.254:36401 requested a full tablet report, sending...
I20260811 18:03:16.466493 31668 ts_manager.cc:194] Registered new tserver with Master: 9fe56ae4a1fb421f9217cdfeaf56c635 (127.27.94.195:33537)
I20260811 18:03:16.467478 28027 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020500763s
I20260811 18:03:16.468238 31668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34452
I20260811 18:03:16.516104 31668 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260811 18:03:16.517891 31668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 18:03:16.585702 31936 tablet_service.cc:1467] Processing CreateTablet for tablet b7a5629fe9d945f0858550c5a1dcf06b (DEFAULT_TABLE table=test-table [id=e3b50493a2a843f2bb7cb26e46c3c860]), partition=
I20260811 18:03:16.587515 31936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a5629fe9d945f0858550c5a1dcf06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:16.591091 31818 tablet_service.cc:1467] Processing CreateTablet for tablet b7a5629fe9d945f0858550c5a1dcf06b (DEFAULT_TABLE table=test-table [id=e3b50493a2a843f2bb7cb26e46c3c860]), partition=
I20260811 18:03:16.592181 31818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a5629fe9d945f0858550c5a1dcf06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:16.592980 32049 tablet_service.cc:1467] Processing CreateTablet for tablet b7a5629fe9d945f0858550c5a1dcf06b (DEFAULT_TABLE table=test-table [id=e3b50493a2a843f2bb7cb26e46c3c860]), partition=
I20260811 18:03:16.594025 32049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a5629fe9d945f0858550c5a1dcf06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:16.611428 32132 tablet_bootstrap.cc:492] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Bootstrap starting.
I20260811 18:03:16.619056 32133 tablet_bootstrap.cc:492] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635: Bootstrap starting.
I20260811 18:03:16.621366 32132 tablet_bootstrap.cc:654] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:16.626890 32134 tablet_bootstrap.cc:492] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Bootstrap starting.
I20260811 18:03:16.627817 32133 tablet_bootstrap.cc:654] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:16.633522 32132 tablet_bootstrap.cc:492] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: No bootstrap required, opened a new log
I20260811 18:03:16.633708 32133 tablet_bootstrap.cc:492] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635: No bootstrap required, opened a new log
I20260811 18:03:16.634135 32133 ts_tablet_manager.cc:1397] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260811 18:03:16.634073 32132 ts_tablet_manager.cc:1397] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.000s
I20260811 18:03:16.634353 32134 tablet_bootstrap.cc:654] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:16.637458 32132 raft_consensus.cc:348] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.638350 32132 raft_consensus.cc:374] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:16.637346 32133 raft_consensus.cc:348] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.638742 32132 raft_consensus.cc:729] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc75d69e2524f61b0d645f74311b9a1, State: Initialized, Role: FOLLOWER
I20260811 18:03:16.640272 32133 raft_consensus.cc:374] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:16.640692 32133 raft_consensus.cc:729] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe56ae4a1fb421f9217cdfeaf56c635, State: Initialized, Role: FOLLOWER
I20260811 18:03:16.641016 32134 tablet_bootstrap.cc:492] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: No bootstrap required, opened a new log
I20260811 18:03:16.640967 32132 consensus_queue.cc:260] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.641753 32134 ts_tablet_manager.cc:1397] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260811 18:03:16.641475 32133 consensus_queue.cc:260] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.643968 31873 heartbeater.cc:503] Master 127.27.94.254:36401 was elected leader, sending a full tablet report...
I20260811 18:03:16.644846 32132 ts_tablet_manager.cc:1428] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260811 18:03:16.645479 32132 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260811 18:03:16.646850 32134 raft_consensus.cc:348] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.648101 32134 raft_consensus.cc:374] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:16.648563 32134 raft_consensus.cc:729] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb7900fae9e44d9bffc6a9d8537996f, State: Initialized, Role: FOLLOWER
I20260811 18:03:16.649628 32134 consensus_queue.cc:260] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.653705 31989 heartbeater.cc:503] Master 127.27.94.254:36401 was elected leader, sending a full tablet report...
I20260811 18:03:16.654316 32133 ts_tablet_manager.cc:1428] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635: Time spent starting tablet: real 0.020s user 0.015s sys 0.005s
I20260811 18:03:16.654316 32134 ts_tablet_manager.cc:1428] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260811 18:03:16.654685 32103 heartbeater.cc:503] Master 127.27.94.254:36401 was elected leader, sending a full tablet report...
W20260811 18:03:16.654958 32133 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
W20260811 18:03:16.655114 32134 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
W20260811 18:03:16.679788 31874 tablet.cc:2369] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 18:03:16.737756 32138 raft_consensus.cc:482] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:03:16.738227 32138 raft_consensus.cc:504] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.740470 32138 leader_election.cc:288] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe56ae4a1fb421f9217cdfeaf56c635 (127.27.94.195:33537), 1cb7900fae9e44d9bffc6a9d8537996f (127.27.94.194:42301)
I20260811 18:03:16.761193 32062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a5629fe9d945f0858550c5a1dcf06b" candidate_uuid: "6bc75d69e2524f61b0d645f74311b9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" is_pre_election: true
I20260811 18:03:16.761241 31949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a5629fe9d945f0858550c5a1dcf06b" candidate_uuid: "6bc75d69e2524f61b0d645f74311b9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" is_pre_election: true
I20260811 18:03:16.762097 32062 raft_consensus.cc:2454] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bc75d69e2524f61b0d645f74311b9a1 in term 0.
I20260811 18:03:16.762691 31949 raft_consensus.cc:2454] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bc75d69e2524f61b0d645f74311b9a1 in term 0.
I20260811 18:03:16.763937 31780 leader_election.cc:302] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc75d69e2524f61b0d645f74311b9a1, 9fe56ae4a1fb421f9217cdfeaf56c635; no voters:
I20260811 18:03:16.764930 32138 raft_consensus.cc:2781] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 18:03:16.765204 32138 raft_consensus.cc:482] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:03:16.765513 32138 raft_consensus.cc:3037] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:16.770968 32138 raft_consensus.cc:504] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.772874 32138 leader_election.cc:288] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [CANDIDATE]: Term 1 election: Requested vote from peers 9fe56ae4a1fb421f9217cdfeaf56c635 (127.27.94.195:33537), 1cb7900fae9e44d9bffc6a9d8537996f (127.27.94.194:42301)
I20260811 18:03:16.773841 32062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a5629fe9d945f0858550c5a1dcf06b" candidate_uuid: "6bc75d69e2524f61b0d645f74311b9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635"
I20260811 18:03:16.774154 31949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7a5629fe9d945f0858550c5a1dcf06b" candidate_uuid: "6bc75d69e2524f61b0d645f74311b9a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb7900fae9e44d9bffc6a9d8537996f"
I20260811 18:03:16.774501 32062 raft_consensus.cc:3037] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:16.774857 31949 raft_consensus.cc:3037] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:16.781378 31949 raft_consensus.cc:2454] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bc75d69e2524f61b0d645f74311b9a1 in term 1.
I20260811 18:03:16.781497 32062 raft_consensus.cc:2454] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bc75d69e2524f61b0d645f74311b9a1 in term 1.
I20260811 18:03:16.783030 31781 leader_election.cc:302] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb7900fae9e44d9bffc6a9d8537996f, 6bc75d69e2524f61b0d645f74311b9a1; no voters:
I20260811 18:03:16.783986 32138 raft_consensus.cc:2781] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:16.785549 32138 raft_consensus.cc:686] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 1 LEADER]: Becoming Leader. State: Replica: 6bc75d69e2524f61b0d645f74311b9a1, State: Running, Role: LEADER
I20260811 18:03:16.786355 32138 consensus_queue.cc:237] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.796360 31666 catalog_manager.cc:5495] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bc75d69e2524f61b0d645f74311b9a1 (127.27.94.193). New cstate: current_term: 1 leader_uuid: "6bc75d69e2524f61b0d645f74311b9a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } health_report { overall_health: UNKNOWN } } }
I20260811 18:03:16.861475 31936 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a5629fe9d945f0858550c5a1dcf06b with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:57514
I20260811 18:03:16.863430 32154 tablet_replica.cc:330] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: stopping tablet replica
I20260811 18:03:16.864158 32154 raft_consensus.cc:2229] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:16.864591 32154 raft_consensus.cc:2258] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:16.866901 32154 ts_tablet_manager.cc:1905] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260811 18:03:16.886271 32154 ts_tablet_manager.cc:1918] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20260811 18:03:16.886924 32154 log.cc:1196] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Deleting WAL directory at /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-1-root/wals/b7a5629fe9d945f0858550c5a1dcf06b
I20260811 18:03:16.892520 32157 ts_tablet_manager.cc:927] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Initiating tablet copy from peer 6bc75d69e2524f61b0d645f74311b9a1 (127.27.94.193:41455)
I20260811 18:03:16.894354 32157 tablet_copy_client.cc:316] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: tablet copy: Beginning tablet copy session from remote peer at address 127.27.94.193:41455
I20260811 18:03:16.908799 31851 tablet_copy_service.cc:140] P 6bc75d69e2524f61b0d645f74311b9a1: Received BeginTabletCopySession request for tablet b7a5629fe9d945f0858550c5a1dcf06b from peer 1cb7900fae9e44d9bffc6a9d8537996f ({username='slave'} at 127.0.0.1:46804)
I20260811 18:03:16.909461 31851 tablet_copy_service.cc:161] P 6bc75d69e2524f61b0d645f74311b9a1: Beginning new tablet copy session on tablet b7a5629fe9d945f0858550c5a1dcf06b from peer 1cb7900fae9e44d9bffc6a9d8537996f at {username='slave'} at 127.0.0.1:46804: session id = 1cb7900fae9e44d9bffc6a9d8537996f-b7a5629fe9d945f0858550c5a1dcf06b
I20260811 18:03:16.916761 31851 tablet_copy_source_session.cc:205] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 18:03:16.920486 32157 ts_tablet_manager.cc:1905] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260811 18:03:16.939227 32157 ts_tablet_manager.cc:1918] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20260811 18:03:16.940685 32157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7a5629fe9d945f0858550c5a1dcf06b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:16.948058 32157 tablet_copy_client.cc:791] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: tablet copy: Starting download of 0 data blocks...
I20260811 18:03:16.948776 32157 tablet_copy_client.cc:655] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: tablet copy: Starting download of 1 WAL segments...
I20260811 18:03:16.950902 31666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34470:
table { table_id: "e3b50493a2a843f2bb7cb26e46c3c860" table_name: "test-table" }
I20260811 18:03:16.951844 31666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34470:
table { table_id: "e3b50493a2a843f2bb7cb26e46c3c860" table_name: "test-table" }
I20260811 18:03:16.954237 32157 tablet_copy_client.cc:531] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 18:03:16.966039 32157 tablet_bootstrap.cc:492] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Bootstrap starting.
I20260811 18:03:16.967041 31666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df: Sending DeleteTablet for 3 replicas of tablet b7a5629fe9d945f0858550c5a1dcf06b
I20260811 18:03:16.969640 32049 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a5629fe9d945f0858550c5a1dcf06b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 18:03:16 UTC) from {username='slave'} at 127.0.0.1:42318
I20260811 18:03:16.970620 31818 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a5629fe9d945f0858550c5a1dcf06b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 18:03:16 UTC) from {username='slave'} at 127.0.0.1:46788
I20260811 18:03:16.970736 31936 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a5629fe9d945f0858550c5a1dcf06b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 18:03:16 UTC) from {username='slave'} at 127.0.0.1:57506
W20260811 18:03:16.973148 31642 catalog_manager.cc:4821] TS 1cb7900fae9e44d9bffc6a9d8537996f (127.27.94.194:42301): delete failed for tablet b7a5629fe9d945f0858550c5a1dcf06b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7a5629fe9d945f0858550c5a1dcf06b already in progress: copying tablet
I20260811 18:03:16.975379 32164 tablet_replica.cc:330] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: stopping tablet replica
I20260811 18:03:16.975975 32163 tablet_replica.cc:330] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635: stopping tablet replica
I20260811 18:03:16.976349 32164 raft_consensus.cc:2229] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:16.977242 32164 pending_rounds.cc:62] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Trying to abort 1 pending ops.
I20260811 18:03:16.977572 32163 raft_consensus.cc:2229] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:16.977586 32164 pending_rounds.cc:68] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317386841245917184 op_type: NO_OP noop_request { }
I20260811 18:03:16.978264 32163 raft_consensus.cc:2258] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:16.978252 32164 raft_consensus.cc:2866] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260811 18:03:16.979090 32164 raft_consensus.cc:2258] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:16.981750 32163 ts_tablet_manager.cc:1905] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 18:03:16.983017 32164 ts_tablet_manager.cc:1905] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 18:03:16.987227 32157 tablet_bootstrap.cc:492] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260811 18:03:16.988422 32157 tablet_bootstrap.cc:492] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Bootstrap complete.
I20260811 18:03:16.989142 32157 ts_tablet_manager.cc:1397] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.008s
I20260811 18:03:16.991919 32157 raft_consensus.cc:348] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.992841 32157 raft_consensus.cc:729] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb7900fae9e44d9bffc6a9d8537996f, State: Initialized, Role: FOLLOWER
I20260811 18:03:16.993575 32157 consensus_queue.cc:260] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe56ae4a1fb421f9217cdfeaf56c635" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 33537 } } peers { permanent_uuid: "6bc75d69e2524f61b0d645f74311b9a1" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 41455 } } peers { permanent_uuid: "1cb7900fae9e44d9bffc6a9d8537996f" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 42301 } }
I20260811 18:03:16.995644 32157 ts_tablet_manager.cc:1428] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260811 18:03:16.995978 32157 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260811 18:03:17.000216 32163 ts_tablet_manager.cc:1918] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260811 18:03:17.000584 31936 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a5629fe9d945f0858550c5a1dcf06b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 18:03:16 UTC) from {username='slave'} at 127.0.0.1:57506
I20260811 18:03:17.000852 32163 log.cc:1196] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635: Deleting WAL directory at /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-2-root/wals/b7a5629fe9d945f0858550c5a1dcf06b
I20260811 18:03:17.001614 32163 ts_tablet_manager.cc:1939] T b7a5629fe9d945f0858550c5a1dcf06b P 9fe56ae4a1fb421f9217cdfeaf56c635: Deleting consensus metadata
I20260811 18:03:17.000221 32164 ts_tablet_manager.cc:1918] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260811 18:03:17.002080 31642 catalog_manager.cc:4821] TS 1cb7900fae9e44d9bffc6a9d8537996f (127.27.94.194:42301): delete failed for tablet b7a5629fe9d945f0858550c5a1dcf06b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7a5629fe9d945f0858550c5a1dcf06b already in progress: copying tablet
I20260811 18:03:17.002228 32164 log.cc:1196] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Deleting WAL directory at /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-0-root/wals/b7a5629fe9d945f0858550c5a1dcf06b
I20260811 18:03:17.003271 32164 ts_tablet_manager.cc:1939] T b7a5629fe9d945f0858550c5a1dcf06b P 6bc75d69e2524f61b0d645f74311b9a1: Deleting consensus metadata
I20260811 18:03:17.005036 31641 catalog_manager.cc:4841] TS 6bc75d69e2524f61b0d645f74311b9a1 (127.27.94.193:41455): tablet b7a5629fe9d945f0858550c5a1dcf06b (table test-table [id=e3b50493a2a843f2bb7cb26e46c3c860]) successfully deleted
I20260811 18:03:17.005784 31641 catalog_manager.cc:4841] TS 9fe56ae4a1fb421f9217cdfeaf56c635 (127.27.94.195:33537): tablet b7a5629fe9d945f0858550c5a1dcf06b (table test-table [id=e3b50493a2a843f2bb7cb26e46c3c860]) successfully deleted
I20260811 18:03:17.048880 31851 tablet_copy_service.cc:342] P 6bc75d69e2524f61b0d645f74311b9a1: Request end of tablet copy session 1cb7900fae9e44d9bffc6a9d8537996f-b7a5629fe9d945f0858550c5a1dcf06b received from {username='slave'} at 127.0.0.1:46804
I20260811 18:03:17.049422 31851 tablet_copy_service.cc:433] P 6bc75d69e2524f61b0d645f74311b9a1: ending tablet copy session 1cb7900fae9e44d9bffc6a9d8537996f-b7a5629fe9d945f0858550c5a1dcf06b on tablet b7a5629fe9d945f0858550c5a1dcf06b with peer 1cb7900fae9e44d9bffc6a9d8537996f
I20260811 18:03:17.052865 31936 tablet_service.cc:1514] Processing DeleteTablet for tablet b7a5629fe9d945f0858550c5a1dcf06b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 18:03:16 UTC) from {username='slave'} at 127.0.0.1:57506
I20260811 18:03:17.053846 32154 tablet_replica.cc:330] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: stopping tablet replica
I20260811 18:03:17.054615 32154 raft_consensus.cc:2229] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:17.055078 32154 pending_rounds.cc:62] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Trying to abort 1 pending ops.
I20260811 18:03:17.055339 32154 pending_rounds.cc:68] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7317386841245917184 op_type: NO_OP noop_request { }
I20260811 18:03:17.055743 32154 raft_consensus.cc:2866] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260811 18:03:17.056174 32154 raft_consensus.cc:2258] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:17.058681 32154 ts_tablet_manager.cc:1905] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 18:03:17.072742 32154 ts_tablet_manager.cc:1918] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260811 18:03:17.073089 32154 log.cc:1196] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Deleting WAL directory at /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1786471338663725-28027-0/minicluster-data/ts-1-root/wals/b7a5629fe9d945f0858550c5a1dcf06b
I20260811 18:03:17.073751 32154 ts_tablet_manager.cc:1939] T b7a5629fe9d945f0858550c5a1dcf06b P 1cb7900fae9e44d9bffc6a9d8537996f: Deleting consensus metadata
I20260811 18:03:17.076464 31642 catalog_manager.cc:4841] TS 1cb7900fae9e44d9bffc6a9d8537996f (127.27.94.194:42301): tablet b7a5629fe9d945f0858550c5a1dcf06b (table test-table [id=e3b50493a2a843f2bb7cb26e46c3c860]) successfully deleted
I20260811 18:03:17.106889 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:03:17.132429 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:17.141855 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:03:17.153915 28027 tablet_server.cc:178] TabletServer@127.27.94.194:0 shutting down...
I20260811 18:03:17.176349 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:17.196249 28027 tablet_server.cc:194] TabletServer@127.27.94.194:0 shutdown complete.
I20260811 18:03:17.207755 28027 tablet_server.cc:178] TabletServer@127.27.94.195:0 shutting down...
I20260811 18:03:17.228451 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:17.247671 28027 tablet_server.cc:194] TabletServer@127.27.94.195:0 shutdown complete.
I20260811 18:03:17.261797 28027 master.cc:537] Master@127.27.94.254:36401 shutting down...
I20260811 18:03:17.288924 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:17.289484 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:17.289808 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3aa40f20a00a44ddbbfc0de40118c4df: stopping tablet replica
I20260811 18:03:17.312938 28027 master.cc:552] Master@127.27.94.254:36401 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1526 ms)
[----------] 4 tests from TsTabletManagerITest (45263 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20260811 18:03:17.343454 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:41989
I20260811 18:03:17.345036 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:17.350468 32179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:17.351647 32180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:17.353185 32182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:17.354035 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:17.354830 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:17.355006 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:17.355171 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471397355159 us; error 0 us; skew 500 ppm
I20260811 18:03:17.355746 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:17.359355 28027 webserver.cc:460] Webserver started at http://127.27.94.254:39087/ using document root <none> and password file <none>
I20260811 18:03:17.359900 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:17.360283 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:17.360615 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:17.362208 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "7708825a790445b4b952e4607bb1f8fc"
format_stamp: "Formatted at 2026-08-11 18:03:17 on dist-test-slave-kvfs"
I20260811 18:03:17.368780 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260811 18:03:17.373744 32189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:17.375020 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260811 18:03:17.375577 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "7708825a790445b4b952e4607bb1f8fc"
format_stamp: "Formatted at 2026-08-11 18:03:17 on dist-test-slave-kvfs"
I20260811 18:03:17.376150 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:17.403126 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:17.404858 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:17.454768 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:41989
I20260811 18:03:17.454861 32269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:41989 every 8 connection(s)
I20260811 18:03:17.461905 32270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:17.474756 32270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc: Bootstrap starting.
I20260811 18:03:17.480861 32270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:17.486774 32270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc: No bootstrap required, opened a new log
I20260811 18:03:17.489315 32270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7708825a790445b4b952e4607bb1f8fc" member_type: VOTER }
I20260811 18:03:17.489749 32270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:17.490006 32270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7708825a790445b4b952e4607bb1f8fc, State: Initialized, Role: FOLLOWER
I20260811 18:03:17.490677 32270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7708825a790445b4b952e4607bb1f8fc" member_type: VOTER }
I20260811 18:03:17.491115 32270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:17.491374 32270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:17.491714 32270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:17.496644 32270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7708825a790445b4b952e4607bb1f8fc" member_type: VOTER }
I20260811 18:03:17.497224 32270 leader_election.cc:302] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7708825a790445b4b952e4607bb1f8fc; no voters:
I20260811 18:03:17.499871 32270 leader_election.cc:288] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:17.500483 32274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:17.502326 32274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 1 LEADER]: Becoming Leader. State: Replica: 7708825a790445b4b952e4607bb1f8fc, State: Running, Role: LEADER
I20260811 18:03:17.503593 32274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7708825a790445b4b952e4607bb1f8fc" member_type: VOTER }
I20260811 18:03:17.505098 32270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:03:17.507534 32278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7708825a790445b4b952e4607bb1f8fc. Latest consensus state: current_term: 1 leader_uuid: "7708825a790445b4b952e4607bb1f8fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7708825a790445b4b952e4607bb1f8fc" member_type: VOTER } }
I20260811 18:03:17.507529 32276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7708825a790445b4b952e4607bb1f8fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7708825a790445b4b952e4607bb1f8fc" member_type: VOTER } }
I20260811 18:03:17.508428 32278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:17.508687 32276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:17.514417 32283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:03:17.524747 32283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:03:17.526479 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:03:17.535177 32283 catalog_manager.cc:1347] Generated new cluster ID: 89a5b9c1f0a246979dc0ba3f79f62dff
I20260811 18:03:17.535524 32283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:03:17.556540 32283 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:03:17.558390 32283 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:03:17.573408 32283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc: Generated new TSK 0
I20260811 18:03:17.574198 32283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:03:17.594867 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:17.601029 32301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:17.602363 32302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:17.604887 28027 server_base.cc:1034] running on GCE node
W20260811 18:03:17.604909 32304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:17.606256 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:17.606449 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:17.606675 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471397606656 us; error 0 us; skew 500 ppm
I20260811 18:03:17.607218 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:17.610605 28027 webserver.cc:460] Webserver started at http://127.27.94.193:44069/ using document root <none> and password file <none>
I20260811 18:03:17.611193 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:17.611420 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:17.611726 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:17.613148 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "88b185c240fa4255b696c06873069fbf"
format_stamp: "Formatted at 2026-08-11 18:03:17 on dist-test-slave-kvfs"
I20260811 18:03:17.619154 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260811 18:03:17.623342 32316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:17.624349 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 18:03:17.624800 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "88b185c240fa4255b696c06873069fbf"
format_stamp: "Formatted at 2026-08-11 18:03:17 on dist-test-slave-kvfs"
I20260811 18:03:17.625162 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:17.640964 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:17.642247 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:17.649308 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:17.649684 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:17.650002 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:17.650240 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:17.723238 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:46673
I20260811 18:03:17.723369 32425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:46673 every 8 connection(s)
I20260811 18:03:17.730520 32426 heartbeater.cc:347] Connected to a master server at 127.27.94.254:41989
I20260811 18:03:17.731019 32426 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:17.731860 32426 heartbeater.cc:511] Master 127.27.94.254:41989 requested a full tablet report, sending...
I20260811 18:03:17.734841 32217 ts_manager.cc:194] Registered new tserver with Master: 88b185c240fa4255b696c06873069fbf (127.27.94.193:46673)
I20260811 18:03:17.735059 28027 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.006972875s
I20260811 18:03:17.737448 32217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59954
I20260811 18:03:17.784427 32374 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260811 18:03:17.785923 32374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:17.801594 32439 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: Bootstrap starting.
I20260811 18:03:17.807842 32439 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:17.812831 32439 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: No bootstrap required, opened a new log
I20260811 18:03:17.813369 32439 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260811 18:03:17.815640 32439 raft_consensus.cc:348] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } }
I20260811 18:03:17.816200 32439 raft_consensus.cc:374] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:17.816485 32439 raft_consensus.cc:729] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b185c240fa4255b696c06873069fbf, State: Initialized, Role: FOLLOWER
I20260811 18:03:17.817044 32439 consensus_queue.cc:260] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } }
I20260811 18:03:17.817605 32439 raft_consensus.cc:388] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:17.817894 32439 raft_consensus.cc:482] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:17.818357 32439 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:17.824585 32439 raft_consensus.cc:504] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } }
I20260811 18:03:17.825354 32439 leader_election.cc:302] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88b185c240fa4255b696c06873069fbf; no voters:
I20260811 18:03:17.827242 32439 leader_election.cc:288] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:17.827560 32441 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:17.829929 32439 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: Time spent starting tablet: real 0.016s user 0.015s sys 0.002s
I20260811 18:03:17.829777 32426 heartbeater.cc:503] Master 127.27.94.254:41989 was elected leader, sending a full tablet report...
I20260811 18:03:17.830112 32441 raft_consensus.cc:686] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 1 LEADER]: Becoming Leader. State: Replica: 88b185c240fa4255b696c06873069fbf, State: Running, Role: LEADER
I20260811 18:03:17.831002 32441 consensus_queue.cc:237] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } }
I20260811 18:03:17.837688 32446 tablet_replica.cc:439] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: TxnStatusTablet state changed. Reason: New leader 88b185c240fa4255b696c06873069fbf. Latest consensus state: current_term: 1 leader_uuid: "88b185c240fa4255b696c06873069fbf" committed_config { opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } } }
I20260811 18:03:17.838568 32446 tablet_replica.cc:442] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: This TxnStatusTablet replica's current role is: LEADER
I20260811 18:03:17.839887 32443 tablet_replica.cc:439] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88b185c240fa4255b696c06873069fbf" committed_config { opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } } }
I20260811 18:03:17.840538 32443 tablet_replica.cc:442] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: This TxnStatusTablet replica's current role is: LEADER
I20260811 18:03:17.841114 32450 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
W20260811 18:03:17.841114 32217 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:17.841794 32450 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260811 18:03:17.846921 32217 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:17.886953 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:03:17.919040 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:17.920259 28027 tablet_replica.cc:330] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: stopping tablet replica
I20260811 18:03:17.921197 28027 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:17.921851 28027 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:17.937908 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:03:17.952437 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:17.959563 32454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:17.960769 32458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:17.963939 32461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:17.964670 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:17.965734 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:17.965958 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:17.966130 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471397966114 us; error 0 us; skew 500 ppm
I20260811 18:03:17.966676 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:17.970585 28027 webserver.cc:460] Webserver started at http://127.27.94.193:44069/ using document root <none> and password file <none>
I20260811 18:03:17.971191 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:17.971411 28027 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260811 18:03:17.976320 28027 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 18:03:17.980543 32466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:17.981736 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 18:03:17.982092 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "88b185c240fa4255b696c06873069fbf"
format_stamp: "Formatted at 2026-08-11 18:03:17 on dist-test-slave-kvfs"
I20260811 18:03:17.982471 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:17.998811 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:17.999948 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:18.011250 32481 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260811 18:03:18.023909 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 18:03:18.024626 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.003s
I20260811 18:03:18.025350 28027 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 18:03:18.033803 28027 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 18:03:18.034116 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.008s sys 0.000s
I20260811 18:03:18.034344 32481 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: Bootstrap starting.
I20260811 18:03:18.098678 32481 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 18:03:18.099992 32481 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: Bootstrap complete.
I20260811 18:03:18.100787 32481 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: Time spent bootstrapping tablet: real 0.067s user 0.041s sys 0.020s
I20260811 18:03:18.103048 32481 raft_consensus.cc:348] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } }
I20260811 18:03:18.103624 32481 raft_consensus.cc:729] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b185c240fa4255b696c06873069fbf, State: Initialized, Role: FOLLOWER
I20260811 18:03:18.104452 32481 consensus_queue.cc:260] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } }
I20260811 18:03:18.105109 32481 raft_consensus.cc:388] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:18.105406 32481 raft_consensus.cc:482] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:18.105731 32481 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:03:18.114275 32481 raft_consensus.cc:504] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } }
I20260811 18:03:18.115191 32481 leader_election.cc:302] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88b185c240fa4255b696c06873069fbf; no voters:
I20260811 18:03:18.123332 32520 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 2 FOLLOWER]: Leader election won for term 2
I20260811 18:03:18.124639 32481 leader_election.cc:288] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 18:03:18.126672 32520 raft_consensus.cc:686] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 2 LEADER]: Becoming Leader. State: Replica: 88b185c240fa4255b696c06873069fbf, State: Running, Role: LEADER
I20260811 18:03:18.127861 32520 consensus_queue.cc:237] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } }
I20260811 18:03:18.129920 32481 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf: Time spent starting tablet: real 0.029s user 0.020s sys 0.005s
I20260811 18:03:18.141722 32534 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "88b185c240fa4255b696c06873069fbf" committed_config { opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } } }
I20260811 18:03:18.141736 32532 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 88b185c240fa4255b696c06873069fbf. Latest consensus state: current_term: 2 leader_uuid: "88b185c240fa4255b696c06873069fbf" committed_config { opid_index: -1 peers { permanent_uuid: "88b185c240fa4255b696c06873069fbf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 46673 } } }
I20260811 18:03:18.142652 32534 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260811 18:03:18.142738 32532 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260811 18:03:18.160550 32546 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 18:03:18.161211 32546 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 18:03:18.188565 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:46673
I20260811 18:03:18.188759 32590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:46673 every 8 connection(s)
I20260811 18:03:18.196856 32591 heartbeater.cc:347] Connected to a master server at 127.27.94.254:41989
I20260811 18:03:18.197363 32591 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:18.198289 32591 heartbeater.cc:511] Master 127.27.94.254:41989 requested a full tablet report, sending...
I20260811 18:03:18.200362 28027 tablet_server.cc:178] TabletServer@127.27.94.193:46673 shutting down...
I20260811 18:03:18.203347 32217 ts_manager.cc:194] Re-registered known tserver with Master: 88b185c240fa4255b696c06873069fbf (127.27.94.193:46673)
W20260811 18:03:18.204301 32217 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:18.207290 32217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59972
I20260811 18:03:18.210579 32591 heartbeater.cc:503] Master 127.27.94.254:41989 was elected leader, sending a full tablet report...
W20260811 18:03:18.214545 32217 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:18.237496 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:18.238755 28027 tablet_replica.cc:330] stopping tablet replica
I20260811 18:03:18.239528 28027 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 2 LEADER]: Raft consensus shutting down.
I20260811 18:03:18.240157 28027 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 88b185c240fa4255b696c06873069fbf [term 2 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:18.266309 28027 tablet_server.cc:194] TabletServer@127.27.94.193:46673 shutdown complete.
I20260811 18:03:18.282871 28027 master.cc:537] Master@127.27.94.254:41989 shutting down...
I20260811 18:03:18.309057 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:18.309801 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:18.310194 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7708825a790445b4b952e4607bb1f8fc: stopping tablet replica
I20260811 18:03:18.331300 28027 master.cc:552] Master@127.27.94.254:41989 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (1020 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20260811 18:03:18.364889 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:42439
I20260811 18:03:18.366128 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:18.372994 32602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:18.374799 32604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:18.376509 32608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:18.378166 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:18.379236 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:18.379477 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:18.379657 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471398379638 us; error 0 us; skew 500 ppm
I20260811 18:03:18.380319 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:18.383903 28027 webserver.cc:460] Webserver started at http://127.27.94.254:37647/ using document root <none> and password file <none>
I20260811 18:03:18.384893 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:18.385306 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:18.385723 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:18.387176 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "882661ebb547464ca88d48ee1ed6ea37"
format_stamp: "Formatted at 2026-08-11 18:03:18 on dist-test-slave-kvfs"
I20260811 18:03:18.392916 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 18:03:18.396999 32617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:18.397984 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 18:03:18.398424 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "882661ebb547464ca88d48ee1ed6ea37"
format_stamp: "Formatted at 2026-08-11 18:03:18 on dist-test-slave-kvfs"
I20260811 18:03:18.398741 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:18.427522 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:18.428730 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:18.494597 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:42439
I20260811 18:03:18.494738 32702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:42439 every 8 connection(s)
I20260811 18:03:18.501138 32703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:18.513640 32703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37: Bootstrap starting.
I20260811 18:03:18.519233 32703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:18.524868 32703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37: No bootstrap required, opened a new log
I20260811 18:03:18.527292 32703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882661ebb547464ca88d48ee1ed6ea37" member_type: VOTER }
I20260811 18:03:18.527741 32703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:18.528013 32703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 882661ebb547464ca88d48ee1ed6ea37, State: Initialized, Role: FOLLOWER
I20260811 18:03:18.528720 32703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882661ebb547464ca88d48ee1ed6ea37" member_type: VOTER }
I20260811 18:03:18.529201 32703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:18.529433 32703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:18.529718 32703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:18.534456 32703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882661ebb547464ca88d48ee1ed6ea37" member_type: VOTER }
I20260811 18:03:18.535055 32703 leader_election.cc:302] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 882661ebb547464ca88d48ee1ed6ea37; no voters:
I20260811 18:03:18.536530 32703 leader_election.cc:288] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:18.536898 32710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:18.538997 32710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 1 LEADER]: Becoming Leader. State: Replica: 882661ebb547464ca88d48ee1ed6ea37, State: Running, Role: LEADER
I20260811 18:03:18.539901 32710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882661ebb547464ca88d48ee1ed6ea37" member_type: VOTER }
I20260811 18:03:18.540612 32703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:03:18.544317 32711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "882661ebb547464ca88d48ee1ed6ea37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882661ebb547464ca88d48ee1ed6ea37" member_type: VOTER } }
I20260811 18:03:18.544915 32711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:18.544591 32712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 882661ebb547464ca88d48ee1ed6ea37. Latest consensus state: current_term: 1 leader_uuid: "882661ebb547464ca88d48ee1ed6ea37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "882661ebb547464ca88d48ee1ed6ea37" member_type: VOTER } }
I20260811 18:03:18.545357 32712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:18.558794 32716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:03:18.566780 32716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:03:18.574196 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:03:18.580474 32716 catalog_manager.cc:1347] Generated new cluster ID: b3020d1be31b4161933d4c3e43be6159
I20260811 18:03:18.580824 32716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:03:18.626089 32716 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:03:18.628968 32716 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:03:18.650243 32716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37: Generated new TSK 0
I20260811 18:03:18.651060 32716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:03:18.710498 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:18.718895 32731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:18.719254 32733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:18.721781 32736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:18.722577 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:18.723847 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:18.724165 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:18.724370 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471398724352 us; error 0 us; skew 500 ppm
I20260811 18:03:18.725131 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:18.729593 28027 webserver.cc:460] Webserver started at http://127.27.94.193:35435/ using document root <none> and password file <none>
I20260811 18:03:18.730325 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:18.730585 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:18.730932 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:18.732620 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "7d9199f000de424c980ac8fa13c37a25"
format_stamp: "Formatted at 2026-08-11 18:03:18 on dist-test-slave-kvfs"
I20260811 18:03:18.739735 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 18:03:18.744527 32748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:18.745518 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 18:03:18.745886 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "7d9199f000de424c980ac8fa13c37a25"
format_stamp: "Formatted at 2026-08-11 18:03:18 on dist-test-slave-kvfs"
I20260811 18:03:18.746277 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:18.769410 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:18.770872 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:18.784310 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:18.784648 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:18.784952 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:18.785169 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:18.880398 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:33083
I20260811 18:03:18.880474 403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:33083 every 8 connection(s)
I20260811 18:03:18.887712 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 18:03:18.888557 404 heartbeater.cc:347] Connected to a master server at 127.27.94.254:42439
I20260811 18:03:18.889060 404 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:18.889871 404 heartbeater.cc:511] Master 127.27.94.254:42439 requested a full tablet report, sending...
I20260811 18:03:18.892067 32645 ts_manager.cc:194] Registered new tserver with Master: 7d9199f000de424c980ac8fa13c37a25 (127.27.94.193:33083)
I20260811 18:03:18.894958 32645 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46684
W20260811 18:03:18.900769 406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:18.899941 407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:18.905319 28027 server_base.cc:1034] running on GCE node
W20260811 18:03:18.906456 411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:18.907750 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:18.907959 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:18.908213 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471398908179 us; error 0 us; skew 500 ppm
I20260811 18:03:18.908777 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:18.911788 28027 webserver.cc:460] Webserver started at http://127.27.94.194:41321/ using document root <none> and password file <none>
I20260811 18:03:18.912400 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:18.912624 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:18.912909 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:18.914124 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-1-root/instance:
uuid: "98d5631ded1c4424b097f4e75f8551bc"
format_stamp: "Formatted at 2026-08-11 18:03:18 on dist-test-slave-kvfs"
I20260811 18:03:18.920677 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 18:03:18.926136 416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:18.927336 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260811 18:03:18.927707 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-1-root
uuid: "98d5631ded1c4424b097f4e75f8551bc"
format_stamp: "Formatted at 2026-08-11 18:03:18 on dist-test-slave-kvfs"
I20260811 18:03:18.928076 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:18.951503 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:18.952885 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:18.962265 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:18.962620 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:18.962915 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:18.963092 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:19.039876 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.194:43331
I20260811 18:03:19.039975 529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.194:43331 every 8 connection(s)
I20260811 18:03:19.045646 534 heartbeater.cc:347] Connected to a master server at 127.27.94.254:42439
I20260811 18:03:19.046144 534 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:19.047127 534 heartbeater.cc:511] Master 127.27.94.254:42439 requested a full tablet report, sending...
I20260811 18:03:19.050248 32645 ts_manager.cc:194] Registered new tserver with Master: 98d5631ded1c4424b097f4e75f8551bc (127.27.94.194:43331)
I20260811 18:03:19.050462 28027 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.006971352s
I20260811 18:03:19.052853 32645 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46704
I20260811 18:03:19.101434 346 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260811 18:03:19.102839 346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:19.118000 547 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: Bootstrap starting.
I20260811 18:03:19.124703 547 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:19.130923 547 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: No bootstrap required, opened a new log
I20260811 18:03:19.131466 547 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260811 18:03:19.134806 547 raft_consensus.cc:348] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } }
I20260811 18:03:19.135685 547 raft_consensus.cc:374] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:19.136000 547 raft_consensus.cc:729] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9199f000de424c980ac8fa13c37a25, State: Initialized, Role: FOLLOWER
I20260811 18:03:19.136698 547 consensus_queue.cc:260] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } }
I20260811 18:03:19.137729 547 raft_consensus.cc:388] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:19.138109 547 raft_consensus.cc:482] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:19.138600 547 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:19.149448 547 raft_consensus.cc:504] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } }
I20260811 18:03:19.150230 547 leader_election.cc:302] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d9199f000de424c980ac8fa13c37a25; no voters:
I20260811 18:03:19.151568 547 leader_election.cc:288] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:19.151880 549 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:19.153601 404 heartbeater.cc:503] Master 127.27.94.254:42439 was elected leader, sending a full tablet report...
I20260811 18:03:19.153820 547 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: Time spent starting tablet: real 0.022s user 0.019s sys 0.005s
I20260811 18:03:19.155090 549 raft_consensus.cc:686] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9199f000de424c980ac8fa13c37a25, State: Running, Role: LEADER
I20260811 18:03:19.155716 549 consensus_queue.cc:237] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } }
I20260811 18:03:19.167934 553 tablet_replica.cc:439] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: TxnStatusTablet state changed. Reason: New leader 7d9199f000de424c980ac8fa13c37a25. Latest consensus state: current_term: 1 leader_uuid: "7d9199f000de424c980ac8fa13c37a25" committed_config { opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } } }
I20260811 18:03:19.168583 553 tablet_replica.cc:442] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: This TxnStatusTablet replica's current role is: LEADER
I20260811 18:03:19.167155 550 tablet_replica.cc:439] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d9199f000de424c980ac8fa13c37a25" committed_config { opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } } }
I20260811 18:03:19.169873 550 tablet_replica.cc:442] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: This TxnStatusTablet replica's current role is: LEADER
W20260811 18:03:19.171252 32645 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:19.176913 559 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 18:03:19.177389 559 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260811 18:03:19.182194 32645 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:19.240090 563 ts_tablet_manager.cc:927] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: Initiating tablet copy from peer 7d9199f000de424c980ac8fa13c37a25 (127.27.94.193:33083)
I20260811 18:03:19.243582 563 tablet_copy_client.cc:316] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: tablet copy: Beginning tablet copy session from remote peer at address 127.27.94.193:33083
I20260811 18:03:19.259331 381 tablet_copy_service.cc:140] P 7d9199f000de424c980ac8fa13c37a25: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 98d5631ded1c4424b097f4e75f8551bc ({username='slave'} at 127.0.0.1:37110)
I20260811 18:03:19.259951 381 tablet_copy_service.cc:161] P 7d9199f000de424c980ac8fa13c37a25: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 98d5631ded1c4424b097f4e75f8551bc at {username='slave'} at 127.0.0.1:37110: session id = 98d5631ded1c4424b097f4e75f8551bc-11111111111111111111111111111111
I20260811 18:03:19.268359 381 tablet_copy_source_session.cc:205] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 18:03:19.271598 563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:19.283753 563 tablet_copy_client.cc:791] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: tablet copy: Starting download of 0 data blocks...
I20260811 18:03:19.284578 563 tablet_copy_client.cc:655] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: tablet copy: Starting download of 1 WAL segments...
I20260811 18:03:19.289466 563 tablet_copy_client.cc:531] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 18:03:19.296653 563 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: Bootstrap starting.
I20260811 18:03:19.326118 563 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 18:03:19.327178 563 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: Bootstrap complete.
I20260811 18:03:19.327920 563 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.004s
I20260811 18:03:19.330515 563 raft_consensus.cc:348] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } }
I20260811 18:03:19.331128 563 raft_consensus.cc:729] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 98d5631ded1c4424b097f4e75f8551bc, State: Initialized, Role: NON_PARTICIPANT
I20260811 18:03:19.331776 563 consensus_queue.cc:260] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } }
I20260811 18:03:19.333866 566 tablet_replica.cc:439] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } } }
I20260811 18:03:19.334122 534 heartbeater.cc:503] Master 127.27.94.254:42439 was elected leader, sending a full tablet report...
I20260811 18:03:19.334524 566 tablet_replica.cc:442] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20260811 18:03:19.334559 563 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 18:03:19.336710 381 tablet_copy_service.cc:342] P 7d9199f000de424c980ac8fa13c37a25: Request end of tablet copy session 98d5631ded1c4424b097f4e75f8551bc-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:37110
I20260811 18:03:19.337356 381 tablet_copy_service.cc:433] P 7d9199f000de424c980ac8fa13c37a25: ending tablet copy session 98d5631ded1c4424b097f4e75f8551bc-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 98d5631ded1c4424b097f4e75f8551bc
W20260811 18:03:19.337867 32645 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260811 18:03:19.343016 32645 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:19.349705 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:03:19.376690 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:19.377983 28027 tablet_replica.cc:330] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: stopping tablet replica
I20260811 18:03:19.378649 28027 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:19.379202 28027 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:19.404892 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:03:19.418802 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:19.425771 568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:19.425967 569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:19.429100 573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:19.431092 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:19.432381 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:19.432643 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:19.432852 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471399432834 us; error 0 us; skew 500 ppm
I20260811 18:03:19.433548 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:19.436636 28027 webserver.cc:460] Webserver started at http://127.27.94.193:35435/ using document root <none> and password file <none>
I20260811 18:03:19.437276 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:19.437485 28027 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260811 18:03:19.442406 28027 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260811 18:03:19.446987 579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:19.448012 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 18:03:19.448400 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "7d9199f000de424c980ac8fa13c37a25"
format_stamp: "Formatted at 2026-08-11 18:03:18 on dist-test-slave-kvfs"
I20260811 18:03:19.448704 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:19.468683 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:19.469790 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:19.478338 592 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260811 18:03:19.489718 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 18:03:19.490072 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.001s
I20260811 18:03:19.490409 28027 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 18:03:19.496886 28027 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 18:03:19.497308 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.006s sys 0.000s
I20260811 18:03:19.498492 592 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: Bootstrap starting.
I20260811 18:03:19.541217 592 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 18:03:19.543015 592 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: Bootstrap complete.
I20260811 18:03:19.545838 592 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: Time spent bootstrapping tablet: real 0.048s user 0.032s sys 0.011s
I20260811 18:03:19.549276 592 raft_consensus.cc:348] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } }
I20260811 18:03:19.550076 592 raft_consensus.cc:729] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9199f000de424c980ac8fa13c37a25, State: Initialized, Role: FOLLOWER
I20260811 18:03:19.551282 592 consensus_queue.cc:260] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } }
I20260811 18:03:19.552307 592 raft_consensus.cc:388] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:19.552664 592 raft_consensus.cc:482] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:19.553061 592 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:03:19.562534 592 raft_consensus.cc:504] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } }
I20260811 18:03:19.563340 592 leader_election.cc:302] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d9199f000de424c980ac8fa13c37a25; no voters:
I20260811 18:03:19.572510 592 leader_election.cc:288] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 18:03:19.574267 616 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 18:03:19.575193 592 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25: Time spent starting tablet: real 0.029s user 0.015s sys 0.009s
I20260811 18:03:19.581053 616 raft_consensus.cc:686] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 2 LEADER]: Becoming Leader. State: Replica: 7d9199f000de424c980ac8fa13c37a25, State: Running, Role: LEADER
I20260811 18:03:19.582531 616 consensus_queue.cc:237] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } }
I20260811 18:03:19.587018 617 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 7d9199f000de424c980ac8fa13c37a25. Latest consensus state: current_term: 2 leader_uuid: "7d9199f000de424c980ac8fa13c37a25" committed_config { opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } } }
I20260811 18:03:19.587702 617 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260811 18:03:19.588941 614 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7d9199f000de424c980ac8fa13c37a25" committed_config { opid_index: -1 peers { permanent_uuid: "7d9199f000de424c980ac8fa13c37a25" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33083 } } }
I20260811 18:03:19.589798 614 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260811 18:03:19.594830 631 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 18:03:19.595350 631 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 18:03:19.647146 711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:33083 every 8 connection(s)
I20260811 18:03:19.647194 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:33083
I20260811 18:03:19.654896 28027 tablet_server.cc:178] TabletServer@127.27.94.193:33083 shutting down...
I20260811 18:03:19.655947 712 heartbeater.cc:347] Connected to a master server at 127.27.94.254:42439
I20260811 18:03:19.658325 712 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:19.659502 712 heartbeater.cc:511] Master 127.27.94.254:42439 requested a full tablet report, sending...
I20260811 18:03:19.664294 32645 ts_manager.cc:194] Re-registered known tserver with Master: 7d9199f000de424c980ac8fa13c37a25 (127.27.94.193:33083)
W20260811 18:03:19.664906 32645 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:19.666751 32645 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46742
I20260811 18:03:19.670851 712 heartbeater.cc:503] Master 127.27.94.254:42439 was elected leader, sending a full tablet report...
W20260811 18:03:19.674572 32645 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:19.695868 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:19.696983 28027 tablet_replica.cc:330] stopping tablet replica
I20260811 18:03:19.699043 28027 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 2 LEADER]: Raft consensus shutting down.
I20260811 18:03:19.699808 28027 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 7d9199f000de424c980ac8fa13c37a25 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:19.724922 28027 tablet_server.cc:194] TabletServer@127.27.94.193:33083 shutdown complete.
I20260811 18:03:19.736123 28027 tablet_server.cc:178] TabletServer@127.27.94.194:0 shutting down...
I20260811 18:03:19.762182 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:19.763114 28027 tablet_replica.cc:330] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc: stopping tablet replica
I20260811 18:03:19.763993 28027 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260811 18:03:19.764839 28027 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 98d5631ded1c4424b097f4e75f8551bc [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260811 18:03:19.787669 28027 tablet_server.cc:194] TabletServer@127.27.94.194:0 shutdown complete.
I20260811 18:03:19.798977 28027 master.cc:537] Master@127.27.94.254:42439 shutting down...
I20260811 18:03:19.829154 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:19.829742 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:19.830055 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P 882661ebb547464ca88d48ee1ed6ea37: stopping tablet replica
I20260811 18:03:19.849865 28027 master.cc:552] Master@127.27.94.254:42439 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1521 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20260811 18:03:19.886647 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:43569
I20260811 18:03:19.888341 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:19.895002 720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:19.895555 721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:19.897863 723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:19.901044 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:19.902379 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:19.902837 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:19.903137 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471399903111 us; error 0 us; skew 500 ppm
I20260811 18:03:19.904191 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:19.908496 28027 webserver.cc:460] Webserver started at http://127.27.94.254:45691/ using document root <none> and password file <none>
I20260811 18:03:19.909101 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:19.909282 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:19.909627 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:19.910881 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "b9094cad59d448b9b46b9b3d22a41692"
format_stamp: "Formatted at 2026-08-11 18:03:19 on dist-test-slave-kvfs"
I20260811 18:03:19.916396 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 18:03:19.921924 729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:19.923156 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260811 18:03:19.923574 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "b9094cad59d448b9b46b9b3d22a41692"
format_stamp: "Formatted at 2026-08-11 18:03:19 on dist-test-slave-kvfs"
I20260811 18:03:19.924327 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:19.969699 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:19.970772 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:20.029196 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:43569
I20260811 18:03:20.029424 799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:43569 every 8 connection(s)
I20260811 18:03:20.033680 801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:20.050782 801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692: Bootstrap starting.
I20260811 18:03:20.055730 801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:20.060608 801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692: No bootstrap required, opened a new log
I20260811 18:03:20.063320 801 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9094cad59d448b9b46b9b3d22a41692" member_type: VOTER }
I20260811 18:03:20.063844 801 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:20.064121 801 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9094cad59d448b9b46b9b3d22a41692, State: Initialized, Role: FOLLOWER
I20260811 18:03:20.064728 801 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9094cad59d448b9b46b9b3d22a41692" member_type: VOTER }
I20260811 18:03:20.065234 801 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:20.065486 801 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:20.065806 801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:20.071409 801 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9094cad59d448b9b46b9b3d22a41692" member_type: VOTER }
I20260811 18:03:20.072093 801 leader_election.cc:302] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9094cad59d448b9b46b9b3d22a41692; no voters:
I20260811 18:03:20.073879 801 leader_election.cc:288] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:20.074373 806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:20.076485 806 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 1 LEADER]: Becoming Leader. State: Replica: b9094cad59d448b9b46b9b3d22a41692, State: Running, Role: LEADER
I20260811 18:03:20.077481 806 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9094cad59d448b9b46b9b3d22a41692" member_type: VOTER }
I20260811 18:03:20.078544 801 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:03:20.082135 808 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9094cad59d448b9b46b9b3d22a41692. Latest consensus state: current_term: 1 leader_uuid: "b9094cad59d448b9b46b9b3d22a41692" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9094cad59d448b9b46b9b3d22a41692" member_type: VOTER } }
I20260811 18:03:20.082183 807 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9094cad59d448b9b46b9b3d22a41692" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9094cad59d448b9b46b9b3d22a41692" member_type: VOTER } }
I20260811 18:03:20.083105 808 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:20.083163 807 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:20.096879 812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:03:20.105242 812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:03:20.108965 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:03:20.117511 812 catalog_manager.cc:1347] Generated new cluster ID: d96725e3aca34b118edd0beba888ec41
I20260811 18:03:20.118108 812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:03:20.185643 812 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:03:20.189426 812 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:03:20.211520 812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692: Generated new TSK 0
I20260811 18:03:20.212524 812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:03:20.248517 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:20.259109 835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:20.259057 836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:20.264966 838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:20.269269 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:20.270534 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:20.270787 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:20.270974 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471400270961 us; error 0 us; skew 500 ppm
I20260811 18:03:20.271622 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:20.276104 28027 webserver.cc:460] Webserver started at http://127.27.94.193:36859/ using document root <none> and password file <none>
I20260811 18:03:20.276779 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:20.277010 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:20.277491 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:20.279482 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "d12efc647feb41c2999938b8640b8eb6"
format_stamp: "Formatted at 2026-08-11 18:03:20 on dist-test-slave-kvfs"
I20260811 18:03:20.285534 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 18:03:20.290987 843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:20.292013 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 18:03:20.292443 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "d12efc647feb41c2999938b8640b8eb6"
format_stamp: "Formatted at 2026-08-11 18:03:20 on dist-test-slave-kvfs"
I20260811 18:03:20.292797 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:20.332964 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:20.334654 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:20.342482 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:20.342871 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:20.343246 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:20.343497 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:20.447780 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:34791
I20260811 18:03:20.448341 955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:34791 every 8 connection(s)
I20260811 18:03:20.455149 956 heartbeater.cc:347] Connected to a master server at 127.27.94.254:43569
I20260811 18:03:20.455711 956 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:20.456679 956 heartbeater.cc:511] Master 127.27.94.254:43569 requested a full tablet report, sending...
I20260811 18:03:20.459474 754 ts_manager.cc:194] Registered new tserver with Master: d12efc647feb41c2999938b8640b8eb6 (127.27.94.193:34791)
I20260811 18:03:20.460379 28027 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.007605892s
I20260811 18:03:20.462845 754 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39532
I20260811 18:03:20.519917 898 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260811 18:03:20.521548 898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:20.537664 969 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6: Bootstrap starting.
I20260811 18:03:20.544611 969 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:20.551419 969 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6: No bootstrap required, opened a new log
I20260811 18:03:20.551990 969 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260811 18:03:20.554895 969 raft_consensus.cc:348] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "d12efc647feb41c2999938b8640b8eb6" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 34791 } }
I20260811 18:03:20.556046 969 raft_consensus.cc:374] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:20.556596 969 raft_consensus.cc:729] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12efc647feb41c2999938b8640b8eb6, State: Initialized, Role: FOLLOWER
I20260811 18:03:20.558105 969 consensus_queue.cc:260] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "d12efc647feb41c2999938b8640b8eb6" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 34791 } }
I20260811 18:03:20.559036 969 raft_consensus.cc:388] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:20.559371 969 raft_consensus.cc:482] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:20.559875 969 raft_consensus.cc:3037] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:20.566787 969 raft_consensus.cc:504] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "d12efc647feb41c2999938b8640b8eb6" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 34791 } }
I20260811 18:03:20.567492 969 leader_election.cc:302] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d12efc647feb41c2999938b8640b8eb6; no voters:
I20260811 18:03:20.567718 973 raft_consensus.cc:482] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:03:20.568284 973 raft_consensus.cc:504] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 peers { permanent_uuid: "d12efc647feb41c2999938b8640b8eb6" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 34791 } }
I20260811 18:03:20.569103 969 leader_election.cc:288] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:20.570111 974 raft_consensus.cc:2781] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:20.572404 973 leader_election.cc:302] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d12efc647feb41c2999938b8640b8eb6; no voters:
I20260811 18:03:20.579746 974 raft_consensus.cc:686] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 1 LEADER]: Becoming Leader. State: Replica: d12efc647feb41c2999938b8640b8eb6, State: Running, Role: LEADER
I20260811 18:03:20.580439 969 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6: Time spent starting tablet: real 0.028s user 0.017s sys 0.009s
I20260811 18:03:20.580891 974 consensus_queue.cc:237] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "d12efc647feb41c2999938b8640b8eb6" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 34791 } }
I20260811 18:03:20.581461 973 leader_election.cc:288] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260811 18:03:20.581912 956 heartbeater.cc:503] Master 127.27.94.254:43569 was elected leader, sending a full tablet report...
W20260811 18:03:20.594117 754 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:20.603437 977 tablet_replica.cc:439] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6: TxnStatusTablet state changed. Reason: New leader d12efc647feb41c2999938b8640b8eb6. Latest consensus state: current_term: 1 leader_uuid: "d12efc647feb41c2999938b8640b8eb6" committed_config { opid_index: -1 peers { permanent_uuid: "d12efc647feb41c2999938b8640b8eb6" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 34791 } } }
I20260811 18:03:20.604522 977 tablet_replica.cc:442] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6: This TxnStatusTablet replica's current role is: LEADER
I20260811 18:03:20.604662 976 tablet_replica.cc:439] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d12efc647feb41c2999938b8640b8eb6" committed_config { opid_index: -1 peers { permanent_uuid: "d12efc647feb41c2999938b8640b8eb6" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 34791 } } }
I20260811 18:03:20.605649 976 tablet_replica.cc:442] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6: This TxnStatusTablet replica's current role is: LEADER
I20260811 18:03:20.606822 982 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 18:03:20.607349 982 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260811 18:03:20.668421 897 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260811 18:03:20.668766 897 txn_status_manager.cc:243] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260811 18:03:20.669095 897 txn_status_manager.cc:205] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20260811 18:03:20.680959 897 txn_status_manager.cc:336] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260811 18:03:20.743002 897 txn_status_manager.cc:243] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260811 18:03:20.743757 897 txn_status_manager.cc:205] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20260811 18:03:20.755471 897 txn_status_manager.cc:336] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20260811 18:03:20.776117 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:56302
W20260811 18:03:20.781018 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:56302
W20260811 18:03:20.785436 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:56302
W20260811 18:03:20.788646 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:56302
W20260811 18:03:20.791687 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:56302
W20260811 18:03:20.794945 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:56302
W20260811 18:03:20.812683 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:56312
W20260811 18:03:20.815188 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:56312
W20260811 18:03:20.817384 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:56312
W20260811 18:03:20.819631 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:56312
W20260811 18:03:20.821997 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:56312
W20260811 18:03:20.824146 898 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:56312
I20260811 18:03:20.829142 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:03:20.857371 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:20.858237 28027 tablet_replica.cc:330] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6: stopping tablet replica
I20260811 18:03:20.858762 28027 raft_consensus.cc:2229] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:20.859195 28027 raft_consensus.cc:2258] T 11111111111111111111111111111111 P d12efc647feb41c2999938b8640b8eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:20.875425 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:03:20.888803 28027 master.cc:537] Master@127.27.94.254:43569 shutting down...
I20260811 18:03:20.914101 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:20.914798 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:20.915131 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P b9094cad59d448b9b46b9b3d22a41692: stopping tablet replica
I20260811 18:03:20.936836 28027 master.cc:552] Master@127.27.94.254:43569 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (1089 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20260811 18:03:20.975793 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:39125
I20260811 18:03:20.977003 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:20.982895 994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:20.984625 996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:20.985177 998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:20.985204 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:20.988839 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:20.989092 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:20.989277 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471400989258 us; error 0 us; skew 500 ppm
I20260811 18:03:20.989820 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:20.992466 28027 webserver.cc:460] Webserver started at http://127.27.94.254:44163/ using document root <none> and password file <none>
I20260811 18:03:20.992964 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:20.993187 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:20.993461 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:20.994693 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "6f4cff6dba194a3c8838f09fb07961f6"
format_stamp: "Formatted at 2026-08-11 18:03:20 on dist-test-slave-kvfs"
I20260811 18:03:21.000690 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260811 18:03:21.004907 1005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:21.005792 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 18:03:21.006175 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "6f4cff6dba194a3c8838f09fb07961f6"
format_stamp: "Formatted at 2026-08-11 18:03:20 on dist-test-slave-kvfs"
I20260811 18:03:21.006524 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:21.017005 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:21.018564 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:21.068292 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:39125
I20260811 18:03:21.068403 1088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:39125 every 8 connection(s)
I20260811 18:03:21.072844 1089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:21.088126 1089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6: Bootstrap starting.
I20260811 18:03:21.093953 1089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:21.098387 1089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6: No bootstrap required, opened a new log
I20260811 18:03:21.100566 1089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4cff6dba194a3c8838f09fb07961f6" member_type: VOTER }
I20260811 18:03:21.100970 1089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:21.101222 1089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4cff6dba194a3c8838f09fb07961f6, State: Initialized, Role: FOLLOWER
I20260811 18:03:21.102105 1089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4cff6dba194a3c8838f09fb07961f6" member_type: VOTER }
I20260811 18:03:21.102664 1089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:21.102892 1089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:21.103183 1089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:21.110535 1089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4cff6dba194a3c8838f09fb07961f6" member_type: VOTER }
I20260811 18:03:21.111240 1089 leader_election.cc:302] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f4cff6dba194a3c8838f09fb07961f6; no voters:
I20260811 18:03:21.113050 1089 leader_election.cc:288] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:21.113623 1092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:21.119031 1092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 1 LEADER]: Becoming Leader. State: Replica: 6f4cff6dba194a3c8838f09fb07961f6, State: Running, Role: LEADER
I20260811 18:03:21.119467 1089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:03:21.120096 1092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4cff6dba194a3c8838f09fb07961f6" member_type: VOTER }
I20260811 18:03:21.124701 1098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f4cff6dba194a3c8838f09fb07961f6. Latest consensus state: current_term: 1 leader_uuid: "6f4cff6dba194a3c8838f09fb07961f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4cff6dba194a3c8838f09fb07961f6" member_type: VOTER } }
I20260811 18:03:21.125372 1098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:21.128494 1097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f4cff6dba194a3c8838f09fb07961f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f4cff6dba194a3c8838f09fb07961f6" member_type: VOTER } }
I20260811 18:03:21.129299 1097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:21.136240 1105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:03:21.144905 1105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:03:21.146235 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:03:21.155098 1105 catalog_manager.cc:1347] Generated new cluster ID: 8f7dcc189ccd48dc8079111bf29e115f
I20260811 18:03:21.155491 1105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:03:21.195688 1105 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:03:21.198832 1105 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:03:21.231331 1105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6: Generated new TSK 0
I20260811 18:03:21.232415 1105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:03:21.278407 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:21.286196 1124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:21.288285 1127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:21.290477 1131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:21.290591 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:21.291759 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:21.292016 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:21.292248 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471401292230 us; error 0 us; skew 500 ppm
I20260811 18:03:21.292754 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:21.295473 28027 webserver.cc:460] Webserver started at http://127.27.94.193:45551/ using document root <none> and password file <none>
I20260811 18:03:21.295943 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:21.296187 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:21.296487 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:21.297727 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "5c5a4fb83d744082816ebcbe397d34bf"
format_stamp: "Formatted at 2026-08-11 18:03:21 on dist-test-slave-kvfs"
I20260811 18:03:21.302981 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 18:03:21.308609 1143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:21.309542 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 18:03:21.309912 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "5c5a4fb83d744082816ebcbe397d34bf"
format_stamp: "Formatted at 2026-08-11 18:03:21 on dist-test-slave-kvfs"
I20260811 18:03:21.310233 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:21.343052 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:21.344367 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:21.353144 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:21.353464 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:21.353750 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:21.353928 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:21.420507 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:33609
I20260811 18:03:21.420666 1238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:33609 every 8 connection(s)
I20260811 18:03:21.426689 1239 heartbeater.cc:347] Connected to a master server at 127.27.94.254:39125
I20260811 18:03:21.427085 1239 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:21.427883 1239 heartbeater.cc:511] Master 127.27.94.254:39125 requested a full tablet report, sending...
I20260811 18:03:21.430182 1032 ts_manager.cc:194] Registered new tserver with Master: 5c5a4fb83d744082816ebcbe397d34bf (127.27.94.193:33609)
I20260811 18:03:21.430344 28027 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.006417957s
I20260811 18:03:21.433223 1032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41238
I20260811 18:03:21.478833 1189 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260811 18:03:21.480285 1189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:21.497701 1259 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf: Bootstrap starting.
I20260811 18:03:21.503187 1259 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:21.508648 1259 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf: No bootstrap required, opened a new log
I20260811 18:03:21.509059 1259 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20260811 18:03:21.511485 1259 raft_consensus.cc:348] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "5c5a4fb83d744082816ebcbe397d34bf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33609 } }
I20260811 18:03:21.511989 1259 raft_consensus.cc:374] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:21.512282 1259 raft_consensus.cc:729] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5a4fb83d744082816ebcbe397d34bf, State: Initialized, Role: FOLLOWER
I20260811 18:03:21.512871 1259 consensus_queue.cc:260] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5c5a4fb83d744082816ebcbe397d34bf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33609 } }
I20260811 18:03:21.513296 1259 raft_consensus.cc:388] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:21.513537 1259 raft_consensus.cc:482] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:21.513780 1259 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:21.519105 1259 raft_consensus.cc:504] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "5c5a4fb83d744082816ebcbe397d34bf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33609 } }
I20260811 18:03:21.519672 1259 leader_election.cc:302] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c5a4fb83d744082816ebcbe397d34bf; no voters:
I20260811 18:03:21.521500 1259 leader_election.cc:288] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:21.522076 1264 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:21.523825 1239 heartbeater.cc:503] Master 127.27.94.254:39125 was elected leader, sending a full tablet report...
I20260811 18:03:21.524121 1259 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260811 18:03:21.524703 1264 raft_consensus.cc:686] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 1 LEADER]: Becoming Leader. State: Replica: 5c5a4fb83d744082816ebcbe397d34bf, State: Running, Role: LEADER
I20260811 18:03:21.525444 1264 consensus_queue.cc:237] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5c5a4fb83d744082816ebcbe397d34bf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33609 } }
I20260811 18:03:21.531208 1266 tablet_replica.cc:439] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf: TxnStatusTablet state changed. Reason: New leader 5c5a4fb83d744082816ebcbe397d34bf. Latest consensus state: current_term: 1 leader_uuid: "5c5a4fb83d744082816ebcbe397d34bf" committed_config { opid_index: -1 peers { permanent_uuid: "5c5a4fb83d744082816ebcbe397d34bf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33609 } } }
I20260811 18:03:21.531854 1266 tablet_replica.cc:442] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf: This TxnStatusTablet replica's current role is: LEADER
I20260811 18:03:21.532298 1265 tablet_replica.cc:439] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c5a4fb83d744082816ebcbe397d34bf" committed_config { opid_index: -1 peers { permanent_uuid: "5c5a4fb83d744082816ebcbe397d34bf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33609 } } }
I20260811 18:03:21.532796 1265 tablet_replica.cc:442] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf: This TxnStatusTablet replica's current role is: LEADER
W20260811 18:03:21.533128 1032 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:21.534654 1271 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 18:03:21.535455 1271 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260811 18:03:21.538833 1032 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260811 18:03:21.566977 1189 tablet_service.cc:1467] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260811 18:03:21.568470 1189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:21.581905 1259 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf: Bootstrap starting.
I20260811 18:03:21.587343 1259 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:21.593931 1259 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf: No bootstrap required, opened a new log
I20260811 18:03:21.594411 1259 ts_tablet_manager.cc:1397] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260811 18:03:21.597363 1259 raft_consensus.cc:348] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "5c5a4fb83d744082816ebcbe397d34bf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33609 } }
I20260811 18:03:21.597903 1259 raft_consensus.cc:374] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:21.598248 1259 raft_consensus.cc:729] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5a4fb83d744082816ebcbe397d34bf, State: Initialized, Role: FOLLOWER
I20260811 18:03:21.598884 1259 consensus_queue.cc:260] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5c5a4fb83d744082816ebcbe397d34bf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33609 } }
I20260811 18:03:21.599402 1259 raft_consensus.cc:388] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:21.599617 1259 raft_consensus.cc:482] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:21.599859 1259 raft_consensus.cc:3037] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:21.605373 1259 raft_consensus.cc:504] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "5c5a4fb83d744082816ebcbe397d34bf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33609 } }
I20260811 18:03:21.605952 1259 leader_election.cc:302] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c5a4fb83d744082816ebcbe397d34bf; no voters:
I20260811 18:03:21.606420 1259 leader_election.cc:288] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:21.606822 1266 raft_consensus.cc:2781] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:21.607406 1266 raft_consensus.cc:686] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 1 LEADER]: Becoming Leader. State: Replica: 5c5a4fb83d744082816ebcbe397d34bf, State: Running, Role: LEADER
I20260811 18:03:21.608330 1266 consensus_queue.cc:237] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "5c5a4fb83d744082816ebcbe397d34bf" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 33609 } }
I20260811 18:03:21.609123 1259 ts_tablet_manager.cc:1428] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf: Time spent starting tablet: real 0.014s user 0.006s sys 0.010s
W20260811 18:03:21.616184 1032 catalog_manager.cc:5328] Ignoring report from unknown tablet 22222222222222222222222222222222
W20260811 18:03:21.621244 1032 catalog_manager.cc:5328] Ignoring report from unknown tablet 22222222222222222222222222222222
I20260811 18:03:21.636860 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:03:21.664213 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:21.665108 28027 tablet_replica.cc:330] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf: stopping tablet replica
I20260811 18:03:21.665702 28027 raft_consensus.cc:2229] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:21.666100 28027 raft_consensus.cc:2258] T 22222222222222222222222222222222 P 5c5a4fb83d744082816ebcbe397d34bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:21.669333 28027 tablet_replica.cc:330] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf: stopping tablet replica
I20260811 18:03:21.669798 28027 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:21.670198 28027 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 5c5a4fb83d744082816ebcbe397d34bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:21.683794 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:03:21.699512 28027 master.cc:537] Master@127.27.94.254:39125 shutting down...
I20260811 18:03:21.729902 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:21.730615 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:21.731082 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6f4cff6dba194a3c8838f09fb07961f6: stopping tablet replica
I20260811 18:03:21.752380 28027 master.cc:552] Master@127.27.94.254:39125 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (805 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (4438 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20260811 18:03:21.781809 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:40083
I20260811 18:03:21.782903 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:21.788892 1284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:21.793367 1285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:21.796208 28027 server_base.cc:1034] running on GCE node
W20260811 18:03:21.796118 1287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:21.797755 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:21.798131 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:21.798362 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471401798336 us; error 0 us; skew 500 ppm
I20260811 18:03:21.798969 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:21.801891 28027 webserver.cc:460] Webserver started at http://127.27.94.254:42673/ using document root <none> and password file <none>
I20260811 18:03:21.802451 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:21.802656 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:21.803002 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:21.804275 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "8abe9edae387427d9c019e2c6092f229"
format_stamp: "Formatted at 2026-08-11 18:03:21 on dist-test-slave-kvfs"
I20260811 18:03:21.811153 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260811 18:03:21.817328 1295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:21.818495 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260811 18:03:21.819051 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "8abe9edae387427d9c019e2c6092f229"
format_stamp: "Formatted at 2026-08-11 18:03:21 on dist-test-slave-kvfs"
I20260811 18:03:21.819437 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:21.837868 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:21.839145 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:21.903800 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:40083
I20260811 18:03:21.903868 1378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:40083 every 8 connection(s)
I20260811 18:03:21.911231 1382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:21.924918 1382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229: Bootstrap starting.
I20260811 18:03:21.929916 1382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:21.934948 1382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229: No bootstrap required, opened a new log
I20260811 18:03:21.937091 1382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abe9edae387427d9c019e2c6092f229" member_type: VOTER }
I20260811 18:03:21.937516 1382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:21.937773 1382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abe9edae387427d9c019e2c6092f229, State: Initialized, Role: FOLLOWER
I20260811 18:03:21.938423 1382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abe9edae387427d9c019e2c6092f229" member_type: VOTER }
I20260811 18:03:21.938908 1382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:21.939285 1382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:21.939787 1382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:21.947417 1382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abe9edae387427d9c019e2c6092f229" member_type: VOTER }
I20260811 18:03:21.948455 1382 leader_election.cc:302] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8abe9edae387427d9c019e2c6092f229; no voters:
I20260811 18:03:21.950223 1382 leader_election.cc:288] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:21.952670 1390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:21.954748 1382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:03:21.954664 1390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 1 LEADER]: Becoming Leader. State: Replica: 8abe9edae387427d9c019e2c6092f229, State: Running, Role: LEADER
I20260811 18:03:21.955826 1390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abe9edae387427d9c019e2c6092f229" member_type: VOTER }
I20260811 18:03:21.960338 1392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8abe9edae387427d9c019e2c6092f229. Latest consensus state: current_term: 1 leader_uuid: "8abe9edae387427d9c019e2c6092f229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abe9edae387427d9c019e2c6092f229" member_type: VOTER } }
I20260811 18:03:21.961265 1392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:21.963145 1391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8abe9edae387427d9c019e2c6092f229" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8abe9edae387427d9c019e2c6092f229" member_type: VOTER } }
I20260811 18:03:21.964107 1391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:21.964092 1398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:03:21.973462 1398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:03:21.981698 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:03:21.989493 1398 catalog_manager.cc:1347] Generated new cluster ID: fb6ec49189f9439994f859d3b6d8c2e7
I20260811 18:03:21.989842 1398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:03:22.017346 1398 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:03:22.020341 1398 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:03:22.039539 1398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229: Generated new TSK 0
I20260811 18:03:22.040357 1398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:03:22.051029 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:22.061432 1414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:22.063476 1415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:22.064502 1420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:22.066226 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:22.067219 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:22.067446 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:22.067831 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471402067789 us; error 0 us; skew 500 ppm
I20260811 18:03:22.069006 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:22.072506 28027 webserver.cc:460] Webserver started at http://127.27.94.193:37399/ using document root <none> and password file <none>
I20260811 18:03:22.073258 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:22.073709 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:22.074204 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:22.076859 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "cdc257cc32694dbdb930659c9a7029a4"
format_stamp: "Formatted at 2026-08-11 18:03:22 on dist-test-slave-kvfs"
I20260811 18:03:22.084368 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260811 18:03:22.089406 1429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:22.090731 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260811 18:03:22.091249 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "cdc257cc32694dbdb930659c9a7029a4"
format_stamp: "Formatted at 2026-08-11 18:03:22 on dist-test-slave-kvfs"
I20260811 18:03:22.091931 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:22.109328 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:22.110550 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:22.112439 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:03:22.115267 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:22.115537 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:22.116216 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:22.116429 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:22.174634 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:45893
I20260811 18:03:22.174847 1533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:45893 every 8 connection(s)
I20260811 18:03:22.180783 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:22.198726 1540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:22.200867 1541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:22.205590 1544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:22.208114 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:22.213394 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:22.214542 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:22.214824 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471402214801 us; error 0 us; skew 500 ppm
I20260811 18:03:22.217491 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:22.220108 1535 heartbeater.cc:347] Connected to a master server at 127.27.94.254:40083
I20260811 18:03:22.220691 1535 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:22.221251 28027 webserver.cc:460] Webserver started at http://127.27.94.194:38317/ using document root <none> and password file <none>
I20260811 18:03:22.221851 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:22.222056 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:22.222334 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:22.222224 1535 heartbeater.cc:511] Master 127.27.94.254:40083 requested a full tablet report, sending...
I20260811 18:03:22.223666 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-1-root/instance:
uuid: "15db42a09b4841e199c612033e1b45a0"
format_stamp: "Formatted at 2026-08-11 18:03:22 on dist-test-slave-kvfs"
I20260811 18:03:22.226847 1324 ts_manager.cc:194] Registered new tserver with Master: cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193:45893)
I20260811 18:03:22.229776 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 18:03:22.230926 1324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48800
I20260811 18:03:22.234884 1554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:22.236182 28027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260811 18:03:22.236992 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-1-root
uuid: "15db42a09b4841e199c612033e1b45a0"
format_stamp: "Formatted at 2026-08-11 18:03:22 on dist-test-slave-kvfs"
I20260811 18:03:22.237422 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:22.258189 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:22.259445 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:22.261229 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:03:22.263808 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:22.264097 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260811 18:03:22.264410 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:22.264551 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:22.316735 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.194:43213
I20260811 18:03:22.316933 1652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.194:43213 every 8 connection(s)
I20260811 18:03:22.322007 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:22.331463 1660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:22.331322 1659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:22.336621 1662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:22.337468 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:22.338635 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:22.338948 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:22.339164 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471402339145 us; error 0 us; skew 500 ppm
I20260811 18:03:22.339934 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:22.341107 1655 heartbeater.cc:347] Connected to a master server at 127.27.94.254:40083
I20260811 18:03:22.341595 1655 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:22.342370 1655 heartbeater.cc:511] Master 127.27.94.254:40083 requested a full tablet report, sending...
I20260811 18:03:22.342928 28027 webserver.cc:460] Webserver started at http://127.27.94.195:38571/ using document root <none> and password file <none>
I20260811 18:03:22.343505 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:22.343856 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:22.344319 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:22.344786 1324 ts_manager.cc:194] Registered new tserver with Master: 15db42a09b4841e199c612033e1b45a0 (127.27.94.194:43213)
I20260811 18:03:22.345853 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-2-root/instance:
uuid: "bb96e4a2c5584241bc27439130df9880"
format_stamp: "Formatted at 2026-08-11 18:03:22 on dist-test-slave-kvfs"
I20260811 18:03:22.346589 1324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48808
I20260811 18:03:22.351943 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 18:03:22.356137 1670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:22.357191 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 18:03:22.357558 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-2-root
uuid: "bb96e4a2c5584241bc27439130df9880"
format_stamp: "Formatted at 2026-08-11 18:03:22 on dist-test-slave-kvfs"
I20260811 18:03:22.357918 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:22.395726 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:22.398634 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:22.401571 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:03:22.407812 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:22.408239 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20260811 18:03:22.408605 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:22.408847 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:22.466437 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.195:44395
I20260811 18:03:22.466589 1774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.195:44395 every 8 connection(s)
I20260811 18:03:22.486755 1776 heartbeater.cc:347] Connected to a master server at 127.27.94.254:40083
I20260811 18:03:22.487254 1776 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:22.488152 1776 heartbeater.cc:511] Master 127.27.94.254:40083 requested a full tablet report, sending...
I20260811 18:03:22.490406 1324 ts_manager.cc:194] Registered new tserver with Master: bb96e4a2c5584241bc27439130df9880 (127.27.94.195:44395)
I20260811 18:03:22.490638 28027 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020742772s
I20260811 18:03:22.493211 1324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48818
I20260811 18:03:22.518748 28027 test_util.cc:268] Using random seed: 1230584003
I20260811 18:03:22.550434 1324 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48828:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 18:03:22.612686 1714 tablet_service.cc:1467] Processing CreateTablet for tablet 0bba212a87084a9996e48d149590e16e (DEFAULT_TABLE table=test-workload [id=bd3bbeb4a3a9495cb347a8d5919db6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 18:03:22.614066 1714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bba212a87084a9996e48d149590e16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:22.622009 1603 tablet_service.cc:1467] Processing CreateTablet for tablet 0bba212a87084a9996e48d149590e16e (DEFAULT_TABLE table=test-workload [id=bd3bbeb4a3a9495cb347a8d5919db6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 18:03:22.623751 1603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bba212a87084a9996e48d149590e16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:22.626195 1476 tablet_service.cc:1467] Processing CreateTablet for tablet 0bba212a87084a9996e48d149590e16e (DEFAULT_TABLE table=test-workload [id=bd3bbeb4a3a9495cb347a8d5919db6e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 18:03:22.627660 1476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bba212a87084a9996e48d149590e16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:22.629782 1808 tablet_bootstrap.cc:492] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880: Bootstrap starting.
I20260811 18:03:22.635288 1808 tablet_bootstrap.cc:654] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:22.647864 1811 tablet_bootstrap.cc:492] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Bootstrap starting.
I20260811 18:03:22.649690 1812 tablet_bootstrap.cc:492] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0: Bootstrap starting.
I20260811 18:03:22.651172 1808 tablet_bootstrap.cc:492] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880: No bootstrap required, opened a new log
I20260811 18:03:22.651664 1808 ts_tablet_manager.cc:1397] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.008s
I20260811 18:03:22.654201 1808 raft_consensus.cc:348] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:22.654877 1808 raft_consensus.cc:374] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:22.655207 1808 raft_consensus.cc:729] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb96e4a2c5584241bc27439130df9880, State: Initialized, Role: FOLLOWER
I20260811 18:03:22.656092 1808 consensus_queue.cc:260] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:22.656868 1811 tablet_bootstrap.cc:654] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:22.657141 1812 tablet_bootstrap.cc:654] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:22.659528 1776 heartbeater.cc:503] Master 127.27.94.254:40083 was elected leader, sending a full tablet report...
I20260811 18:03:22.660006 1808 ts_tablet_manager.cc:1428] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260811 18:03:22.664254 1811 tablet_bootstrap.cc:492] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: No bootstrap required, opened a new log
I20260811 18:03:22.664294 1812 tablet_bootstrap.cc:492] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0: No bootstrap required, opened a new log
I20260811 18:03:22.664857 1811 ts_tablet_manager.cc:1397] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20260811 18:03:22.664943 1812 ts_tablet_manager.cc:1397] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20260811 18:03:22.667389 1811 raft_consensus.cc:348] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:22.668314 1811 raft_consensus.cc:374] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:22.667918 1812 raft_consensus.cc:348] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:22.668622 1811 raft_consensus.cc:729] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc257cc32694dbdb930659c9a7029a4, State: Initialized, Role: FOLLOWER
I20260811 18:03:22.668784 1812 raft_consensus.cc:374] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:22.669162 1812 raft_consensus.cc:729] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15db42a09b4841e199c612033e1b45a0, State: Initialized, Role: FOLLOWER
I20260811 18:03:22.669698 1811 consensus_queue.cc:260] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:22.669860 1812 consensus_queue.cc:260] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:22.673370 1535 heartbeater.cc:503] Master 127.27.94.254:40083 was elected leader, sending a full tablet report...
I20260811 18:03:22.673983 1811 ts_tablet_manager.cc:1428] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Time spent starting tablet: real 0.009s user 0.002s sys 0.007s
I20260811 18:03:22.677402 1655 heartbeater.cc:503] Master 127.27.94.254:40083 was elected leader, sending a full tablet report...
I20260811 18:03:22.677951 1812 ts_tablet_manager.cc:1428] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260811 18:03:22.788473 1813 raft_consensus.cc:482] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:03:22.789201 1813 raft_consensus.cc:504] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:22.791752 1813 leader_election.cc:288] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15db42a09b4841e199c612033e1b45a0 (127.27.94.194:43213), cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193:45893)
I20260811 18:03:22.804597 1617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bba212a87084a9996e48d149590e16e" candidate_uuid: "bb96e4a2c5584241bc27439130df9880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15db42a09b4841e199c612033e1b45a0" is_pre_election: true
I20260811 18:03:22.805465 1617 raft_consensus.cc:2454] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb96e4a2c5584241bc27439130df9880 in term 0.
I20260811 18:03:22.806627 1675 leader_election.cc:302] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15db42a09b4841e199c612033e1b45a0, bb96e4a2c5584241bc27439130df9880; no voters:
I20260811 18:03:22.807386 1813 raft_consensus.cc:2781] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 18:03:22.807739 1813 raft_consensus.cc:482] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:03:22.807484 1494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bba212a87084a9996e48d149590e16e" candidate_uuid: "bb96e4a2c5584241bc27439130df9880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc257cc32694dbdb930659c9a7029a4" is_pre_election: true
I20260811 18:03:22.808094 1813 raft_consensus.cc:3037] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:22.808363 1494 raft_consensus.cc:2454] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb96e4a2c5584241bc27439130df9880 in term 0.
I20260811 18:03:22.813398 1813 raft_consensus.cc:504] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:22.815294 1813 leader_election.cc:288] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [CANDIDATE]: Term 1 election: Requested vote from peers 15db42a09b4841e199c612033e1b45a0 (127.27.94.194:43213), cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193:45893)
I20260811 18:03:22.816480 1494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bba212a87084a9996e48d149590e16e" candidate_uuid: "bb96e4a2c5584241bc27439130df9880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdc257cc32694dbdb930659c9a7029a4"
I20260811 18:03:22.816493 1617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bba212a87084a9996e48d149590e16e" candidate_uuid: "bb96e4a2c5584241bc27439130df9880" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15db42a09b4841e199c612033e1b45a0"
I20260811 18:03:22.817138 1494 raft_consensus.cc:3037] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:22.817215 1617 raft_consensus.cc:3037] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:22.824155 1494 raft_consensus.cc:2454] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb96e4a2c5584241bc27439130df9880 in term 1.
I20260811 18:03:22.825264 1617 raft_consensus.cc:2454] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb96e4a2c5584241bc27439130df9880 in term 1.
I20260811 18:03:22.825601 1674 leader_election.cc:302] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb96e4a2c5584241bc27439130df9880, cdc257cc32694dbdb930659c9a7029a4; no voters:
I20260811 18:03:22.826858 1813 raft_consensus.cc:2781] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:22.828483 1813 raft_consensus.cc:686] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 1 LEADER]: Becoming Leader. State: Replica: bb96e4a2c5584241bc27439130df9880, State: Running, Role: LEADER
I20260811 18:03:22.829368 1813 consensus_queue.cc:237] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:22.838023 1322 catalog_manager.cc:5495] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb96e4a2c5584241bc27439130df9880 (127.27.94.195). New cstate: current_term: 1 leader_uuid: "bb96e4a2c5584241bc27439130df9880" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } } }
I20260811 18:03:23.168272 1603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:23.168741 1714 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 18:03:23.171404 1476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:23.234373 1813 consensus_queue.cc:1035] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 18:03:23.300890 1813 consensus_queue.cc:1035] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8abe9edae387427d9c019e2c6092f229 | 127.27.94.254:40083 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
15db42a09b4841e199c612033e1b45a0 | 127.27.94.194:43213 | HEALTHY | <none> | 0 | 0
bb96e4a2c5584241bc27439130df9880 | 127.27.94.195:44395 | HEALTHY | <none> | 1 | 0
cdc257cc32694dbdb930659c9a7029a4 | 127.27.94.193:45893 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 0bba212a87084a9996e48d149590e16e of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
bb96e4a2c5584241bc27439130df9880 (127.27.94.195:44395): RUNNING [LEADER]
15db42a09b4841e199c612033e1b45a0 (127.27.94.194:43213): RUNNING
cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193:45893): RUNNING
All reported replicas are:
A = bb96e4a2c5584241bc27439130df9880
B = 15db42a09b4841e199c612033e1b45a0
C = cdc257cc32694dbdb930659c9a7029a4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 18:03:23.452823 28027 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 18:03:23.894142 1714 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 18:03:23.898871 1476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:23.899114 1603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8abe9edae387427d9c019e2c6092f229 | 127.27.94.254:40083 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
15db42a09b4841e199c612033e1b45a0 | 127.27.94.194:43213 | HEALTHY | <none> | 0 | 0
bb96e4a2c5584241bc27439130df9880 | 127.27.94.195:44395 | HEALTHY | <none> | 1 | 0
cdc257cc32694dbdb930659c9a7029a4 | 127.27.94.193:45893 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260811 18:03:24.147557 28027 log_verifier.cc:126] Checking tablet 0bba212a87084a9996e48d149590e16e
I20260811 18:03:24.228005 28027 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0bba212a87084a9996e48d149590e16e
I20260811 18:03:24.228729 28027 tablet_replica.cc:330] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: stopping tablet replica
I20260811 18:03:24.229331 28027 raft_consensus.cc:2229] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:24.229885 28027 raft_consensus.cc:2258] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260811 18:03:24.388666 1674 consensus_peers.cc:487] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 -> Peer cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193:45893): Couldn't send request to peer cdc257cc32694dbdb930659c9a7029a4. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260811 18:03:24.389922 1813 raft_consensus.cc:998] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880: Attempting to remove follower cdc257cc32694dbdb930659c9a7029a4 from the Raft config. Reason: The tablet replica hosted on peer cdc257cc32694dbdb930659c9a7029a4 has failed
I20260811 18:03:24.392719 1813 consensus_queue.cc:237] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } }
I20260811 18:03:24.402608 1616 raft_consensus.cc:1261] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 1 FOLLOWER]: Refusing update from remote peer bb96e4a2c5584241bc27439130df9880: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 18:03:24.403995 1932 consensus_queue.cc:1035] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260811 18:03:24.411949 1813 raft_consensus.cc:2932] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } }
I20260811 18:03:24.413619 1616 raft_consensus.cc:2932] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } }
I20260811 18:03:24.426980 1321 catalog_manager.cc:5495] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 reported cstate change: config changed from index -1 to 2, VOTER cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193) evicted. New cstate: current_term: 1 leader_uuid: "bb96e4a2c5584241bc27439130df9880" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } }
I20260811 18:03:24.444968 1476 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bba212a87084a9996e48d149590e16e with delete_type TABLET_DATA_TOMBSTONED (TS cdc257cc32694dbdb930659c9a7029a4 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:35052
I20260811 18:03:24.447261 1955 ts_tablet_manager.cc:1905] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260811 18:03:24.450404 1734 consensus_queue.cc:237] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
W20260811 18:03:24.458871 1674 consensus_peers.cc:487] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 -> Peer cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193:45893): Couldn't send request to peer cdc257cc32694dbdb930659c9a7029a4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260811 18:03:24.459380 1616 raft_consensus.cc:1261] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 1 FOLLOWER]: Refusing update from remote peer bb96e4a2c5584241bc27439130df9880: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260811 18:03:24.461539 1932 consensus_queue.cc:1035] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260811 18:03:24.466042 1955 ts_tablet_manager.cc:1918] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260811 18:03:24.466531 1955 log.cc:1196] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Deleting WAL directory at /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1786471338663725-28027-0/minicluster-data/ts-0-root/wals/0bba212a87084a9996e48d149590e16e
I20260811 18:03:24.468663 1302 catalog_manager.cc:4841] TS cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193:45893): tablet 0bba212a87084a9996e48d149590e16e (table test-workload [id=bd3bbeb4a3a9495cb347a8d5919db6e5]) successfully deleted
I20260811 18:03:24.468575 1813 raft_consensus.cc:2932] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } } }
I20260811 18:03:24.478745 1297 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0bba212a87084a9996e48d149590e16e with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260811 18:03:24.478654 1616 raft_consensus.cc:2932] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } } }
I20260811 18:03:24.483108 1320 catalog_manager.cc:5495] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 reported cstate change: config changed from index 2 to 3, VOTER cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193) added. New cstate: current_term: 1 leader_uuid: "bb96e4a2c5584241bc27439130df9880" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } } }
I20260811 18:03:24.533183 1603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:24.546518 1714 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 18:03:24.546739 1476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8abe9edae387427d9c019e2c6092f229 | 127.27.94.254:40083 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
15db42a09b4841e199c612033e1b45a0 | 127.27.94.194:43213 | HEALTHY | <none> | 0 | 0
bb96e4a2c5584241bc27439130df9880 | 127.27.94.195:44395 | HEALTHY | <none> | 1 | 0
cdc257cc32694dbdb930659c9a7029a4 | 127.27.94.193:45893 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 0bba212a87084a9996e48d149590e16e of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
bb96e4a2c5584241bc27439130df9880 (127.27.94.195:44395): RUNNING [LEADER]
15db42a09b4841e199c612033e1b45a0 (127.27.94.194:43213): RUNNING
All reported replicas are:
A = bb96e4a2c5584241bc27439130df9880
B = 15db42a09b4841e199c612033e1b45a0
C = cdc257cc32694dbdb930659c9a7029a4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 2
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 18:03:24.769356 28027 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 18:03:24.879154 1972 ts_tablet_manager.cc:927] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Initiating tablet copy from peer bb96e4a2c5584241bc27439130df9880 (127.27.94.195:44395)
I20260811 18:03:24.880707 1972 tablet_copy_client.cc:316] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: tablet copy: Beginning tablet copy session from remote peer at address 127.27.94.195:44395
I20260811 18:03:24.892798 1749 tablet_copy_service.cc:140] P bb96e4a2c5584241bc27439130df9880: Received BeginTabletCopySession request for tablet 0bba212a87084a9996e48d149590e16e from peer cdc257cc32694dbdb930659c9a7029a4 ({username='slave'} at 127.0.0.1:40552)
I20260811 18:03:24.893460 1749 tablet_copy_service.cc:161] P bb96e4a2c5584241bc27439130df9880: Beginning new tablet copy session on tablet 0bba212a87084a9996e48d149590e16e from peer cdc257cc32694dbdb930659c9a7029a4 at {username='slave'} at 127.0.0.1:40552: session id = cdc257cc32694dbdb930659c9a7029a4-0bba212a87084a9996e48d149590e16e
I20260811 18:03:24.899554 1749 tablet_copy_source_session.cc:205] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 18:03:24.902132 1972 ts_tablet_manager.cc:1905] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260811 18:03:24.913935 1972 ts_tablet_manager.cc:1918] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260811 18:03:24.914637 1972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bba212a87084a9996e48d149590e16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:24.920239 1972 tablet_copy_client.cc:791] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: tablet copy: Starting download of 0 data blocks...
I20260811 18:03:24.920817 1972 tablet_copy_client.cc:655] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: tablet copy: Starting download of 1 WAL segments...
I20260811 18:03:24.925482 1972 tablet_copy_client.cc:531] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 18:03:24.934201 1972 tablet_bootstrap.cc:492] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Bootstrap starting.
I20260811 18:03:24.956612 1972 tablet_bootstrap.cc:492] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 18:03:24.957458 1972 tablet_bootstrap.cc:492] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Bootstrap complete.
I20260811 18:03:24.958180 1972 ts_tablet_manager.cc:1397] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.011s
I20260811 18:03:24.960871 1972 raft_consensus.cc:348] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:24.961539 1972 raft_consensus.cc:729] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc257cc32694dbdb930659c9a7029a4, State: Initialized, Role: FOLLOWER
I20260811 18:03:24.962289 1972 consensus_queue.cc:260] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bb96e4a2c5584241bc27439130df9880" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 44395 } } peers { permanent_uuid: "15db42a09b4841e199c612033e1b45a0" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 43213 } } peers { permanent_uuid: "cdc257cc32694dbdb930659c9a7029a4" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45893 } }
I20260811 18:03:24.965258 1972 ts_tablet_manager.cc:1428] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: Time spent starting tablet: real 0.007s user 0.003s sys 0.005s
I20260811 18:03:24.967520 1749 tablet_copy_service.cc:342] P bb96e4a2c5584241bc27439130df9880: Request end of tablet copy session cdc257cc32694dbdb930659c9a7029a4-0bba212a87084a9996e48d149590e16e received from {username='slave'} at 127.0.0.1:40552
I20260811 18:03:24.967941 1749 tablet_copy_service.cc:433] P bb96e4a2c5584241bc27439130df9880: ending tablet copy session cdc257cc32694dbdb930659c9a7029a4-0bba212a87084a9996e48d149590e16e on tablet 0bba212a87084a9996e48d149590e16e with peer cdc257cc32694dbdb930659c9a7029a4
I20260811 18:03:25.188091 1603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:25.202865 1476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:25.203790 1714 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 18:03:25.263303 1494 raft_consensus.cc:1203] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8abe9edae387427d9c019e2c6092f229 | 127.27.94.254:40083 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
15db42a09b4841e199c612033e1b45a0 | 127.27.94.194:43213 | HEALTHY | <none> | 0 | 0
bb96e4a2c5584241bc27439130df9880 | 127.27.94.195:44395 | HEALTHY | <none> | 1 | 0
cdc257cc32694dbdb930659c9a7029a4 | 127.27.94.193:45893 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 0bba212a87084a9996e48d149590e16e of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
bb96e4a2c5584241bc27439130df9880 (127.27.94.195:44395): RUNNING [LEADER]
15db42a09b4841e199c612033e1b45a0 (127.27.94.194:43213): RUNNING
cdc257cc32694dbdb930659c9a7029a4 (127.27.94.193:45893): RUNNING
All reported replicas are:
A = bb96e4a2c5584241bc27439130df9880
B = 15db42a09b4841e199c612033e1b45a0
C = cdc257cc32694dbdb930659c9a7029a4
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | A B C | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 18:03:25.438915 28027 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 18:03:25.921556 1603 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:25.931149 1476 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:25.931828 1714 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8abe9edae387427d9c019e2c6092f229 | 127.27.94.254:40083 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
15db42a09b4841e199c612033e1b45a0 | 127.27.94.194:43213 | HEALTHY | <none> | 0 | 0
bb96e4a2c5584241bc27439130df9880 | 127.27.94.195:44395 | HEALTHY | <none> | 1 | 0
cdc257cc32694dbdb930659c9a7029a4 | 127.27.94.193:45893 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260811 18:03:26.153563 28027 log_verifier.cc:126] Checking tablet 0bba212a87084a9996e48d149590e16e
I20260811 18:03:26.228118 28027 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0bba212a87084a9996e48d149590e16e
I20260811 18:03:26.230608 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:03:26.260331 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:26.261031 28027 tablet_replica.cc:330] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4: stopping tablet replica
I20260811 18:03:26.261687 28027 raft_consensus.cc:2229] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:26.262123 28027 raft_consensus.cc:2258] T 0bba212a87084a9996e48d149590e16e P cdc257cc32694dbdb930659c9a7029a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:26.284811 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:03:26.297398 28027 tablet_server.cc:178] TabletServer@127.27.94.194:0 shutting down...
I20260811 18:03:26.324872 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:26.325553 28027 tablet_replica.cc:330] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0: stopping tablet replica
I20260811 18:03:26.326133 28027 raft_consensus.cc:2229] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:26.326706 28027 raft_consensus.cc:2258] T 0bba212a87084a9996e48d149590e16e P 15db42a09b4841e199c612033e1b45a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:26.348904 28027 tablet_server.cc:194] TabletServer@127.27.94.194:0 shutdown complete.
I20260811 18:03:26.359689 28027 tablet_server.cc:178] TabletServer@127.27.94.195:0 shutting down...
I20260811 18:03:26.389487 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:26.390179 28027 tablet_replica.cc:330] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880: stopping tablet replica
I20260811 18:03:26.390812 28027 raft_consensus.cc:2229] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:26.391685 28027 raft_consensus.cc:2258] T 0bba212a87084a9996e48d149590e16e P bb96e4a2c5584241bc27439130df9880 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:26.415434 28027 tablet_server.cc:194] TabletServer@127.27.94.195:0 shutdown complete.
I20260811 18:03:26.426997 28027 master.cc:537] Master@127.27.94.254:40083 shutting down...
I20260811 18:03:26.452291 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:26.452816 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:26.453076 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8abe9edae387427d9c019e2c6092f229: stopping tablet replica
I20260811 18:03:26.473977 28027 master.cc:552] Master@127.27.94.254:40083 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (4726 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20260811 18:03:26.508941 28027 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.94.254:36017
I20260811 18:03:26.510100 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:26.516314 2071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:26.517720 2072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:26.519923 2075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:26.520893 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:26.521692 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:26.521869 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:26.521986 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471406521974 us; error 0 us; skew 500 ppm
I20260811 18:03:26.522436 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:26.525365 28027 webserver.cc:460] Webserver started at http://127.27.94.254:43227/ using document root <none> and password file <none>
I20260811 18:03:26.525849 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:26.526005 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:26.526211 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:26.527320 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/master-0-root/instance:
uuid: "f5e200813e36468093d3fab3b5731c94"
format_stamp: "Formatted at 2026-08-11 18:03:26 on dist-test-slave-kvfs"
I20260811 18:03:26.532593 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 18:03:26.537227 2082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:26.538317 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 18:03:26.538637 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/master-0-root
uuid: "f5e200813e36468093d3fab3b5731c94"
format_stamp: "Formatted at 2026-08-11 18:03:26 on dist-test-slave-kvfs"
I20260811 18:03:26.538919 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:26.569896 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:26.570935 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:26.622809 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.254:36017
I20260811 18:03:26.622924 2170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.254:36017 every 8 connection(s)
I20260811 18:03:26.627667 2172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:26.642205 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94: Bootstrap starting.
I20260811 18:03:26.646525 2172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:26.650810 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94: No bootstrap required, opened a new log
I20260811 18:03:26.652858 2172 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e200813e36468093d3fab3b5731c94" member_type: VOTER }
I20260811 18:03:26.653287 2172 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:26.653460 2172 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e200813e36468093d3fab3b5731c94, State: Initialized, Role: FOLLOWER
I20260811 18:03:26.653949 2172 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e200813e36468093d3fab3b5731c94" member_type: VOTER }
I20260811 18:03:26.654350 2172 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 18:03:26.654533 2172 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 18:03:26.654743 2172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:26.659348 2172 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e200813e36468093d3fab3b5731c94" member_type: VOTER }
I20260811 18:03:26.660010 2172 leader_election.cc:302] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5e200813e36468093d3fab3b5731c94; no voters:
I20260811 18:03:26.661381 2172 leader_election.cc:288] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 18:03:26.661755 2177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:26.663340 2177 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 1 LEADER]: Becoming Leader. State: Replica: f5e200813e36468093d3fab3b5731c94, State: Running, Role: LEADER
I20260811 18:03:26.664134 2177 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e200813e36468093d3fab3b5731c94" member_type: VOTER }
I20260811 18:03:26.664415 2172 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 18:03:26.669299 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5e200813e36468093d3fab3b5731c94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e200813e36468093d3fab3b5731c94" member_type: VOTER } }
I20260811 18:03:26.669997 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:26.669234 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5e200813e36468093d3fab3b5731c94. Latest consensus state: current_term: 1 leader_uuid: "f5e200813e36468093d3fab3b5731c94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e200813e36468093d3fab3b5731c94" member_type: VOTER } }
I20260811 18:03:26.670405 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [sys.catalog]: This master's current role is: LEADER
I20260811 18:03:26.674254 2192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 18:03:26.680384 2192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 18:03:26.684232 28027 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260811 18:03:26.690186 2192 catalog_manager.cc:1347] Generated new cluster ID: b2e7297c38964865b575f59a29ca8bcc
I20260811 18:03:26.690546 2192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 18:03:26.716507 2192 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 18:03:26.718711 2192 catalog_manager.cc:1504] Loading token signing keys...
I20260811 18:03:26.733312 2192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94: Generated new TSK 0
I20260811 18:03:26.734216 2192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 18:03:26.751845 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:26.759124 2205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:26.760550 2208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:26.762984 28027 server_base.cc:1034] running on GCE node
W20260811 18:03:26.762917 2212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:26.764201 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:26.764446 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:26.764654 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471406764635 us; error 0 us; skew 500 ppm
I20260811 18:03:26.765476 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:26.768507 28027 webserver.cc:460] Webserver started at http://127.27.94.193:34679/ using document root <none> and password file <none>
I20260811 18:03:26.769039 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:26.769304 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:26.769649 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:26.770835 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-0-root/instance:
uuid: "f57c0256cce14bb3a2eb87d3f12fa164"
format_stamp: "Formatted at 2026-08-11 18:03:26 on dist-test-slave-kvfs"
I20260811 18:03:26.776165 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260811 18:03:26.781006 2220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:26.781988 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260811 18:03:26.782353 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-0-root
uuid: "f57c0256cce14bb3a2eb87d3f12fa164"
format_stamp: "Formatted at 2026-08-11 18:03:26 on dist-test-slave-kvfs"
I20260811 18:03:26.782657 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:26.799762 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:26.801203 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:26.802963 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:03:26.805927 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:26.806143 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:26.806452 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:26.806605 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:26.863173 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.193:45985
I20260811 18:03:26.863427 2314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.193:45985 every 8 connection(s)
I20260811 18:03:26.869130 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:26.879303 2320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:26.880154 2321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:26.881704 2326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:26.884148 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:26.885746 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:26.885993 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:26.886202 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471406886184 us; error 0 us; skew 500 ppm
I20260811 18:03:26.886920 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:26.889607 2315 heartbeater.cc:347] Connected to a master server at 127.27.94.254:36017
I20260811 18:03:26.890029 2315 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:26.890509 28027 webserver.cc:460] Webserver started at http://127.27.94.194:46085/ using document root <none> and password file <none>
I20260811 18:03:26.890897 2315 heartbeater.cc:511] Master 127.27.94.254:36017 requested a full tablet report, sending...
I20260811 18:03:26.891124 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:26.891420 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:26.891737 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:26.893309 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-1-root/instance:
uuid: "17a4d3b2ce0f4459843cfaa371a5f03c"
format_stamp: "Formatted at 2026-08-11 18:03:26 on dist-test-slave-kvfs"
I20260811 18:03:26.893800 2114 ts_manager.cc:194] Registered new tserver with Master: f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193:45985)
I20260811 18:03:26.896351 2114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41366
I20260811 18:03:26.899206 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 18:03:26.903312 2336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:26.904498 28027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 18:03:26.904852 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-1-root
uuid: "17a4d3b2ce0f4459843cfaa371a5f03c"
format_stamp: "Formatted at 2026-08-11 18:03:26 on dist-test-slave-kvfs"
I20260811 18:03:26.905220 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:26.915851 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:26.917136 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:26.918618 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:03:26.921947 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:26.922163 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:26.922492 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:26.922639 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:26.974747 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.194:45023
I20260811 18:03:26.974943 2433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.194:45023 every 8 connection(s)
I20260811 18:03:26.980248 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:26.988328 2441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:26.989373 2442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:26.993676 2446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:26.993813 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:26.995030 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:26.995302 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:26.995518 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471406995498 us; error 0 us; skew 500 ppm
I20260811 18:03:26.996304 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:26.998060 2436 heartbeater.cc:347] Connected to a master server at 127.27.94.254:36017
I20260811 18:03:26.998474 2436 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:26.999351 2436 heartbeater.cc:511] Master 127.27.94.254:36017 requested a full tablet report, sending...
I20260811 18:03:26.999500 28027 webserver.cc:460] Webserver started at http://127.27.94.195:35109/ using document root <none> and password file <none>
I20260811 18:03:27.000217 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:27.000452 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:27.001012 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:27.001729 2114 ts_manager.cc:194] Registered new tserver with Master: 17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194:45023)
I20260811 18:03:27.002538 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-2-root/instance:
uuid: "962bcd545c534f88a3770241aa87c3cf"
format_stamp: "Formatted at 2026-08-11 18:03:26 on dist-test-slave-kvfs"
I20260811 18:03:27.003646 2114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41372
I20260811 18:03:27.007973 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 18:03:27.011948 2453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:27.012944 28027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 18:03:27.013370 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-2-root
uuid: "962bcd545c534f88a3770241aa87c3cf"
format_stamp: "Formatted at 2026-08-11 18:03:26 on dist-test-slave-kvfs"
I20260811 18:03:27.013732 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:27.033408 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:27.034701 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:27.036513 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:03:27.038988 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:27.039196 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:27.039467 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:27.039615 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:27.090488 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.195:34313
I20260811 18:03:27.090715 2548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.195:34313 every 8 connection(s)
I20260811 18:03:27.096357 28027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 18:03:27.104727 2558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:27.105731 2560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 18:03:27.111449 2562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 18:03:27.111840 28027 server_base.cc:1034] running on GCE node
I20260811 18:03:27.113255 28027 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260811 18:03:27.113559 28027 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260811 18:03:27.114238 28027 hybrid_clock.cc:648] HybridClock initialized: now 1786471407114203 us; error 0 us; skew 500 ppm
I20260811 18:03:27.115253 28027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 18:03:27.115442 2550 heartbeater.cc:347] Connected to a master server at 127.27.94.254:36017
I20260811 18:03:27.116278 2550 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:27.117412 2550 heartbeater.cc:511] Master 127.27.94.254:36017 requested a full tablet report, sending...
I20260811 18:03:27.118882 28027 webserver.cc:460] Webserver started at http://127.27.94.196:43877/ using document root <none> and password file <none>
I20260811 18:03:27.119643 28027 fs_manager.cc:359] Metadata directory not provided
I20260811 18:03:27.119854 28027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 18:03:27.120195 28027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 18:03:27.120234 2114 ts_manager.cc:194] Registered new tserver with Master: 962bcd545c534f88a3770241aa87c3cf (127.27.94.195:34313)
I20260811 18:03:27.121418 28027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-3-root/instance:
uuid: "adcc9b7c422d4ee28ba4acd7a32c634b"
format_stamp: "Formatted at 2026-08-11 18:03:27 on dist-test-slave-kvfs"
I20260811 18:03:27.122586 2114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41382
I20260811 18:03:27.127614 28027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260811 18:03:27.134516 2568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:27.136236 28027 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260811 18:03:27.136943 28027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-3-root
uuid: "adcc9b7c422d4ee28ba4acd7a32c634b"
format_stamp: "Formatted at 2026-08-11 18:03:27 on dist-test-slave-kvfs"
I20260811 18:03:27.137583 28027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 18:03:27.161096 28027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 18:03:27.162683 28027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 18:03:27.164578 28027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 18:03:27.167930 28027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 18:03:27.168299 28027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260811 18:03:27.168566 28027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 18:03:27.168740 28027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 18:03:27.222426 28027 rpc_server.cc:307] RPC server started. Bound to: 127.27.94.196:33743
I20260811 18:03:27.222680 2662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.94.196:33743 every 8 connection(s)
I20260811 18:03:27.258432 2663 heartbeater.cc:347] Connected to a master server at 127.27.94.254:36017
I20260811 18:03:27.260169 2663 heartbeater.cc:464] Registering TS with master...
I20260811 18:03:27.261421 2663 heartbeater.cc:511] Master 127.27.94.254:36017 requested a full tablet report, sending...
I20260811 18:03:27.264755 2114 ts_manager.cc:194] Registered new tserver with Master: adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743)
I20260811 18:03:27.264885 28027 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.038660715s
I20260811 18:03:27.267271 2114 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41396
I20260811 18:03:27.293313 28027 test_util.cc:268] Using random seed: 1235358567
I20260811 18:03:27.327706 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41422:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 18:03:27.395002 2498 tablet_service.cc:1467] Processing CreateTablet for tablet 6525f6df4a774a8dba636d7b43d9f3f2 (DEFAULT_TABLE table=test-workload [id=b3ad8a2a6f1b4f2cbb99cf010ff61e85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 18:03:27.396442 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6525f6df4a774a8dba636d7b43d9f3f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:27.404366 2377 tablet_service.cc:1467] Processing CreateTablet for tablet 6525f6df4a774a8dba636d7b43d9f3f2 (DEFAULT_TABLE table=test-workload [id=b3ad8a2a6f1b4f2cbb99cf010ff61e85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 18:03:27.405030 2613 tablet_service.cc:1467] Processing CreateTablet for tablet 6525f6df4a774a8dba636d7b43d9f3f2 (DEFAULT_TABLE table=test-workload [id=b3ad8a2a6f1b4f2cbb99cf010ff61e85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 18:03:27.406023 2377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6525f6df4a774a8dba636d7b43d9f3f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:27.406447 2613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6525f6df4a774a8dba636d7b43d9f3f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:27.419275 2697 tablet_bootstrap.cc:492] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf: Bootstrap starting.
I20260811 18:03:27.426966 2698 tablet_bootstrap.cc:492] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c: Bootstrap starting.
I20260811 18:03:27.432397 2697 tablet_bootstrap.cc:654] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:27.433679 2698 tablet_bootstrap.cc:654] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:27.442250 2698 tablet_bootstrap.cc:492] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c: No bootstrap required, opened a new log
I20260811 18:03:27.442797 2698 ts_tablet_manager.cc:1397] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260811 18:03:27.446223 2697 tablet_bootstrap.cc:492] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf: No bootstrap required, opened a new log
I20260811 18:03:27.445919 2698 raft_consensus.cc:348] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:27.447176 2697 ts_tablet_manager.cc:1397] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.008s
I20260811 18:03:27.448530 2704 tablet_bootstrap.cc:492] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b: Bootstrap starting.
I20260811 18:03:27.447265 2698 raft_consensus.cc:374] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:27.449440 2698 raft_consensus.cc:729] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a4d3b2ce0f4459843cfaa371a5f03c, State: Initialized, Role: FOLLOWER
I20260811 18:03:27.450212 2698 consensus_queue.cc:260] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:27.450541 2697 raft_consensus.cc:348] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:27.453974 2697 raft_consensus.cc:374] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:27.454499 2697 raft_consensus.cc:729] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962bcd545c534f88a3770241aa87c3cf, State: Initialized, Role: FOLLOWER
I20260811 18:03:27.457146 2697 consensus_queue.cc:260] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:27.460423 2436 heartbeater.cc:503] Master 127.27.94.254:36017 was elected leader, sending a full tablet report...
I20260811 18:03:27.460883 2698 ts_tablet_manager.cc:1428] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260811 18:03:27.461891 2550 heartbeater.cc:503] Master 127.27.94.254:36017 was elected leader, sending a full tablet report...
I20260811 18:03:27.462467 2697 ts_tablet_manager.cc:1428] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260811 18:03:27.460474 2704 tablet_bootstrap.cc:654] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b: Neither blocks nor log segments found. Creating new log.
I20260811 18:03:27.472544 2704 tablet_bootstrap.cc:492] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b: No bootstrap required, opened a new log
I20260811 18:03:27.473373 2704 ts_tablet_manager.cc:1397] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.009s
I20260811 18:03:27.475584 2704 raft_consensus.cc:348] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:27.476475 2704 raft_consensus.cc:374] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 18:03:27.476765 2704 raft_consensus.cc:729] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcc9b7c422d4ee28ba4acd7a32c634b, State: Initialized, Role: FOLLOWER
I20260811 18:03:27.477466 2704 consensus_queue.cc:260] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:27.480994 2704 ts_tablet_manager.cc:1428] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260811 18:03:27.481158 2663 heartbeater.cc:503] Master 127.27.94.254:36017 was elected leader, sending a full tablet report...
I20260811 18:03:27.522367 2716 raft_consensus.cc:482] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 18:03:27.523088 2716 raft_consensus.cc:504] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:27.525542 2716 leader_election.cc:288] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194:45023), 962bcd545c534f88a3770241aa87c3cf (127.27.94.195:34313)
I20260811 18:03:27.542982 2389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6525f6df4a774a8dba636d7b43d9f3f2" candidate_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" is_pre_election: true
I20260811 18:03:27.544390 2389 raft_consensus.cc:2454] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adcc9b7c422d4ee28ba4acd7a32c634b in term 0.
I20260811 18:03:27.546141 2570 leader_election.cc:302] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a4d3b2ce0f4459843cfaa371a5f03c, adcc9b7c422d4ee28ba4acd7a32c634b; no voters:
I20260811 18:03:27.546669 2514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6525f6df4a774a8dba636d7b43d9f3f2" candidate_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962bcd545c534f88a3770241aa87c3cf" is_pre_election: true
I20260811 18:03:27.547117 2716 raft_consensus.cc:2781] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 18:03:27.547498 2716 raft_consensus.cc:482] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 18:03:27.547528 2514 raft_consensus.cc:2454] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adcc9b7c422d4ee28ba4acd7a32c634b in term 0.
I20260811 18:03:27.548360 2716 raft_consensus.cc:3037] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:27.554531 2716 raft_consensus.cc:504] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:27.556425 2716 leader_election.cc:288] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [CANDIDATE]: Term 1 election: Requested vote from peers 17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194:45023), 962bcd545c534f88a3770241aa87c3cf (127.27.94.195:34313)
I20260811 18:03:27.557539 2389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6525f6df4a774a8dba636d7b43d9f3f2" candidate_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c"
I20260811 18:03:27.557590 2514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6525f6df4a774a8dba636d7b43d9f3f2" candidate_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962bcd545c534f88a3770241aa87c3cf"
I20260811 18:03:27.558450 2389 raft_consensus.cc:3037] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:27.558735 2514 raft_consensus.cc:3037] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 0 FOLLOWER]: Advancing to term 1
I20260811 18:03:27.563922 2389 raft_consensus.cc:2454] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adcc9b7c422d4ee28ba4acd7a32c634b in term 1.
I20260811 18:03:27.564733 2514 raft_consensus.cc:2454] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adcc9b7c422d4ee28ba4acd7a32c634b in term 1.
I20260811 18:03:27.565464 2570 leader_election.cc:302] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a4d3b2ce0f4459843cfaa371a5f03c, adcc9b7c422d4ee28ba4acd7a32c634b; no voters:
I20260811 18:03:27.566493 2716 raft_consensus.cc:2781] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 18:03:27.567853 2716 raft_consensus.cc:686] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 1 LEADER]: Becoming Leader. State: Replica: adcc9b7c422d4ee28ba4acd7a32c634b, State: Running, Role: LEADER
I20260811 18:03:27.568748 2716 consensus_queue.cc:237] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:27.576639 2114 catalog_manager.cc:5495] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b reported cstate change: term changed from 0 to 1, leader changed from <none> to adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196). New cstate: current_term: 1 leader_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } health_report { overall_health: HEALTHY } } }
I20260811 18:03:27.886404 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:27.900231 2315 heartbeater.cc:503] Master 127.27.94.254:36017 was elected leader, sending a full tablet report...
I20260811 18:03:27.900403 2498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:27.902176 2266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:27.905340 2613 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 18:03:28.151404 2774 consensus_queue.cc:1035] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 18:03:28.189193 2774 consensus_queue.cc:1035] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [LEADER]: Connected to new peer: Peer: permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
f5e200813e36468093d3fab3b5731c94 | 127.27.94.254:36017 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
17a4d3b2ce0f4459843cfaa371a5f03c | 127.27.94.194:45023 | HEALTHY | <none> | 0 | 0
962bcd545c534f88a3770241aa87c3cf | 127.27.94.195:34313 | HEALTHY | <none> | 0 | 0
adcc9b7c422d4ee28ba4acd7a32c634b | 127.27.94.196:33743 | HEALTHY | <none> | 1 | 0
f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 6525f6df4a774a8dba636d7b43d9f3f2 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194:45023): RUNNING
962bcd545c534f88a3770241aa87c3cf (127.27.94.195:34313): RUNNING
adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743): RUNNING [LEADER]
All reported replicas are:
A = 17a4d3b2ce0f4459843cfaa371a5f03c
B = 962bcd545c534f88a3770241aa87c3cf
C = adcc9b7c422d4ee28ba4acd7a32c634b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 18:03:28.236778 28027 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 18:03:28.674113 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:28.684047 2498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:28.692013 2266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:28.698812 2613 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
f5e200813e36468093d3fab3b5731c94 | 127.27.94.254:36017 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
17a4d3b2ce0f4459843cfaa371a5f03c | 127.27.94.194:45023 | HEALTHY | <none> | 0 | 0
962bcd545c534f88a3770241aa87c3cf | 127.27.94.195:34313 | HEALTHY | <none> | 0 | 0
adcc9b7c422d4ee28ba4acd7a32c634b | 127.27.94.196:33743 | HEALTHY | <none> | 1 | 0
f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260811 18:03:28.950420 28027 log_verifier.cc:126] Checking tablet 6525f6df4a774a8dba636d7b43d9f3f2
I20260811 18:03:29.035079 28027 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6525f6df4a774a8dba636d7b43d9f3f2
I20260811 18:03:29.035606 28027 tablet_replica.cc:330] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b: stopping tablet replica
I20260811 18:03:29.036722 28027 raft_consensus.cc:2229] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:29.037602 28027 raft_consensus.cc:2258] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:29.318243 2498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:29.321220 2613 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:29.322156 2266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:29.323649 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
f5e200813e36468093d3fab3b5731c94 | 127.27.94.254:36017 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
17a4d3b2ce0f4459843cfaa371a5f03c | 127.27.94.194:45023 | HEALTHY | <none> | 0 | 0
962bcd545c534f88a3770241aa87c3cf | 127.27.94.195:34313 | HEALTHY | <none> | 0 | 0
adcc9b7c422d4ee28ba4acd7a32c634b | 127.27.94.196:33743 | HEALTHY | <none> | 0 | 0
f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 6525f6df4a774a8dba636d7b43d9f3f2 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194:45023): RUNNING
962bcd545c534f88a3770241aa87c3cf (127.27.94.195:34313): RUNNING
adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = 17a4d3b2ce0f4459843cfaa371a5f03c
B = 962bcd545c534f88a3770241aa87c3cf
C = adcc9b7c422d4ee28ba4acd7a32c634b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 18:03:29.570767 28027 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 18:03:30.001344 2377 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:30.004884 2498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:30.018015 2266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:30.019860 2613 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:30.105564 2917 raft_consensus.cc:482] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader adcc9b7c422d4ee28ba4acd7a32c634b)
I20260811 18:03:30.106402 2917 raft_consensus.cc:504] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:30.135625 2917 leader_election.cc:288] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194:45023), adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743)
I20260811 18:03:30.182818 2389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6525f6df4a774a8dba636d7b43d9f3f2" candidate_uuid: "962bcd545c534f88a3770241aa87c3cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" is_pre_election: true
I20260811 18:03:30.190032 2625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6525f6df4a774a8dba636d7b43d9f3f2" candidate_uuid: "962bcd545c534f88a3770241aa87c3cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" is_pre_election: true
W20260811 18:03:30.191608 2454 leader_election.cc:341] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743): Illegal state: cannot vote while shut down
I20260811 18:03:30.192198 2454 leader_election.cc:302] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 962bcd545c534f88a3770241aa87c3cf; no voters: 17a4d3b2ce0f4459843cfaa371a5f03c, adcc9b7c422d4ee28ba4acd7a32c634b
I20260811 18:03:30.193236 2917 raft_consensus.cc:2726] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 18:03:30.296504 2921 raft_consensus.cc:482] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader adcc9b7c422d4ee28ba4acd7a32c634b)
I20260811 18:03:30.298044 2921 raft_consensus.cc:504] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:30.326038 2921 leader_election.cc:288] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 962bcd545c534f88a3770241aa87c3cf (127.27.94.195:34313), adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743)
I20260811 18:03:30.362524 2514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6525f6df4a774a8dba636d7b43d9f3f2" candidate_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "962bcd545c534f88a3770241aa87c3cf" is_pre_election: true
I20260811 18:03:30.363847 2514 raft_consensus.cc:2454] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17a4d3b2ce0f4459843cfaa371a5f03c in term 1.
I20260811 18:03:30.365410 2337 leader_election.cc:302] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17a4d3b2ce0f4459843cfaa371a5f03c, 962bcd545c534f88a3770241aa87c3cf; no voters:
I20260811 18:03:30.366335 2921 raft_consensus.cc:2781] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 18:03:30.366710 2921 raft_consensus.cc:482] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 1 FOLLOWER]: Starting leader election (detected failure of leader adcc9b7c422d4ee28ba4acd7a32c634b)
I20260811 18:03:30.367046 2921 raft_consensus.cc:3037] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 1 FOLLOWER]: Advancing to term 2
I20260811 18:03:30.373373 2625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6525f6df4a774a8dba636d7b43d9f3f2" candidate_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" is_pre_election: true
W20260811 18:03:30.375473 2337 leader_election.cc:341] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743): Illegal state: cannot vote while shut down
I20260811 18:03:30.380136 2921 raft_consensus.cc:504] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
I20260811 18:03:30.382989 2921 leader_election.cc:288] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [CANDIDATE]: Term 2 election: Requested vote from peers 962bcd545c534f88a3770241aa87c3cf (127.27.94.195:34313), adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743)
I20260811 18:03:30.383522 2514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6525f6df4a774a8dba636d7b43d9f3f2" candidate_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "962bcd545c534f88a3770241aa87c3cf"
I20260811 18:03:30.384076 2625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6525f6df4a774a8dba636d7b43d9f3f2" candidate_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b"
I20260811 18:03:30.384517 2514 raft_consensus.cc:3037] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 1 FOLLOWER]: Advancing to term 2
W20260811 18:03:30.385607 2337 leader_election.cc:341] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743): Illegal state: cannot vote while shut down
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
f5e200813e36468093d3fab3b5731c94 | 127.27.94.254:36017 | HEALTHY
Unusual flags for Master:
I20260811 18:03:30.394141 2514 raft_consensus.cc:2454] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17a4d3b2ce0f4459843cfaa371a5f03c in term 2.
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
I20260811 18:03:30.395496 2337 leader_election.cc:302] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17a4d3b2ce0f4459843cfaa371a5f03c, 962bcd545c534f88a3770241aa87c3cf; no voters: adcc9b7c422d4ee28ba4acd7a32c634b
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
17a4d3b2ce0f4459843cfaa371a5f03c | 127.27.94.194:45023 | HEALTHY | <none> | 0 | 0
962bcd545c534f88a3770241aa87c3cf | 127.27.94.195:34313 | HEALTHY | <none> | 0 | 0
adcc9b7c422d4ee28ba4acd7a32c634b | 127.27.94.196:33743 | HEALTHY | <none> | 0 | 0
f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
I20260811 18:03:30.396435 2921 raft_consensus.cc:2781] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 2 FOLLOWER]: Leader election won for term 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 6525f6df4a774a8dba636d7b43d9f3f2 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194:45023): RUNNING
962bcd545c534f88a3770241aa87c3cf (127.27.94.195:34313): RUNNING
adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743): not running [LEADER]
I20260811 18:03:30.397687 2921 raft_consensus.cc:686] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 2 LEADER]: Becoming Leader. State: Replica: 17a4d3b2ce0f4459843cfaa371a5f03c, State: Running, Role: LEADER
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = 17a4d3b2ce0f4459843cfaa371a5f03c
B = 962bcd545c534f88a3770241aa87c3cf
C = adcc9b7c422d4ee28ba4acd7a32c634b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 1 | -1 | Yes
I20260811 18:03:30.399061 2921 consensus_queue.cc:237] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } }
B | A B C* | 1 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 18:03:30.416535 28027 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 18:03:30.417657 2113 catalog_manager.cc:5495] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c reported cstate change: term changed from 1 to 2, leader changed from adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196) to 17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194). New cstate: current_term: 2 leader_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adcc9b7c422d4ee28ba4acd7a32c634b" member_type: VOTER last_known_addr { host: "127.27.94.196" port: 33743 } health_report { overall_health: UNKNOWN } } }
I20260811 18:03:30.941241 2377 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 18:03:30.961092 2266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:30.972427 2613 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:30.978085 2498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:31.008955 2337 consensus_queue.cc:579] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [LEADER]: The tablet replica hosted on peer adcc9b7c422d4ee28ba4acd7a32c634b has failed
W20260811 18:03:31.021534 2337 consensus_peers.cc:487] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c -> Peer adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743): Couldn't send request to peer adcc9b7c422d4ee28ba4acd7a32c634b. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260811 18:03:31.030663 2514 raft_consensus.cc:1261] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 2 FOLLOWER]: Refusing update from remote peer 17a4d3b2ce0f4459843cfaa371a5f03c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260811 18:03:31.040076 2981 consensus_queue.cc:1035] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260811 18:03:31.108006 2389 consensus_queue.cc:237] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } }
I20260811 18:03:31.125197 2514 raft_consensus.cc:1261] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 2 FOLLOWER]: Refusing update from remote peer 17a4d3b2ce0f4459843cfaa371a5f03c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260811 18:03:31.127348 2981 consensus_queue.cc:1035] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260811 18:03:31.136209 2982 raft_consensus.cc:2932] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } }
I20260811 18:03:31.149197 2514 raft_consensus.cc:2932] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } }
I20260811 18:03:31.158986 2087 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6525f6df4a774a8dba636d7b43d9f3f2 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260811 18:03:31.165952 2109 catalog_manager.cc:5495] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c reported cstate change: config changed from index -1 to 3, VOTER adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196) evicted. New cstate: current_term: 2 leader_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } health_report { overall_health: HEALTHY } } }
I20260811 18:03:31.199308 2613 tablet_service.cc:1514] Processing DeleteTablet for tablet 6525f6df4a774a8dba636d7b43d9f3f2 with delete_type TABLET_DATA_TOMBSTONED (TS adcc9b7c422d4ee28ba4acd7a32c634b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54316
I20260811 18:03:31.203097 2389 consensus_queue.cc:237] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: NON_VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: true } }
I20260811 18:03:31.230310 2514 raft_consensus.cc:1261] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 2 FOLLOWER]: Refusing update from remote peer 17a4d3b2ce0f4459843cfaa371a5f03c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260811 18:03:31.233709 2982 consensus_queue.cc:1035] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260811 18:03:31.244148 2981 raft_consensus.cc:2932] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: NON_VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: true } } }
I20260811 18:03:31.255422 2514 raft_consensus.cc:2932] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: NON_VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: true } } }
W20260811 18:03:31.267189 2339 consensus_peers.cc:487] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c -> Peer f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193:45985): Couldn't send request to peer f57c0256cce14bb3a2eb87d3f12fa164. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6525f6df4a774a8dba636d7b43d9f3f2. This is attempt 1: this message will repeat every 5th retry.
I20260811 18:03:31.275554 2087 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6525f6df4a774a8dba636d7b43d9f3f2 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260811 18:03:31.277679 2997 ts_tablet_manager.cc:1905] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260811 18:03:31.282737 2109 catalog_manager.cc:5495] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c reported cstate change: config changed from index 3 to 4, NON_VOTER f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193) added. New cstate: current_term: 2 leader_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: NON_VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260811 18:03:31.301203 2997 ts_tablet_manager.cc:1918] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260811 18:03:31.301776 2997 log.cc:1196] T 6525f6df4a774a8dba636d7b43d9f3f2 P adcc9b7c422d4ee28ba4acd7a32c634b: Deleting WAL directory at /tmp/dist-test-taskbnQUm1/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1786471338663725-28027-0/minicluster-data/ts-3-root/wals/6525f6df4a774a8dba636d7b43d9f3f2
I20260811 18:03:31.304113 2083 catalog_manager.cc:4841] TS adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743): tablet 6525f6df4a774a8dba636d7b43d9f3f2 (table test-workload [id=b3ad8a2a6f1b4f2cbb99cf010ff61e85]) successfully deleted
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
f5e200813e36468093d3fab3b5731c94 | 127.27.94.254:36017 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
17a4d3b2ce0f4459843cfaa371a5f03c | 127.27.94.194:45023 | HEALTHY | <none> | 1 | 0
962bcd545c534f88a3770241aa87c3cf | 127.27.94.195:34313 | HEALTHY | <none> | 0 | 0
adcc9b7c422d4ee28ba4acd7a32c634b | 127.27.94.196:33743 | HEALTHY | <none> | 0 | 0
f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 6525f6df4a774a8dba636d7b43d9f3f2 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194:45023): RUNNING [LEADER]
962bcd545c534f88a3770241aa87c3cf (127.27.94.195:34313): RUNNING
adcc9b7c422d4ee28ba4acd7a32c634b (127.27.94.196:33743): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
All reported replicas are:
A = 17a4d3b2ce0f4459843cfaa371a5f03c
B = 962bcd545c534f88a3770241aa87c3cf
C = adcc9b7c422d4ee28ba4acd7a32c634b
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | A B C | 2 | -1 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 18:03:31.378459 28027 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 18:03:31.736200 3010 ts_tablet_manager.cc:927] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: Initiating tablet copy from peer 17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194:45023)
I20260811 18:03:31.738548 3010 tablet_copy_client.cc:316] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: tablet copy: Beginning tablet copy session from remote peer at address 127.27.94.194:45023
I20260811 18:03:31.756515 2409 tablet_copy_service.cc:140] P 17a4d3b2ce0f4459843cfaa371a5f03c: Received BeginTabletCopySession request for tablet 6525f6df4a774a8dba636d7b43d9f3f2 from peer f57c0256cce14bb3a2eb87d3f12fa164 ({username='slave'} at 127.0.0.1:48816)
I20260811 18:03:31.757144 2409 tablet_copy_service.cc:161] P 17a4d3b2ce0f4459843cfaa371a5f03c: Beginning new tablet copy session on tablet 6525f6df4a774a8dba636d7b43d9f3f2 from peer f57c0256cce14bb3a2eb87d3f12fa164 at {username='slave'} at 127.0.0.1:48816: session id = f57c0256cce14bb3a2eb87d3f12fa164-6525f6df4a774a8dba636d7b43d9f3f2
I20260811 18:03:31.765625 2409 tablet_copy_source_session.cc:205] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 18:03:31.769649 3010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6525f6df4a774a8dba636d7b43d9f3f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 18:03:31.785344 3010 tablet_copy_client.cc:791] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: tablet copy: Starting download of 0 data blocks...
I20260811 18:03:31.786005 3010 tablet_copy_client.cc:655] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: tablet copy: Starting download of 1 WAL segments...
I20260811 18:03:31.790431 3010 tablet_copy_client.cc:531] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 18:03:31.799981 3010 tablet_bootstrap.cc:492] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: Bootstrap starting.
I20260811 18:03:31.827180 3010 tablet_bootstrap.cc:492] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 18:03:31.828197 3010 tablet_bootstrap.cc:492] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: Bootstrap complete.
I20260811 18:03:31.828850 3010 ts_tablet_manager.cc:1397] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.012s
I20260811 18:03:31.832378 3010 raft_consensus.cc:348] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: NON_VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: true } }
I20260811 18:03:31.833375 3010 raft_consensus.cc:729] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f57c0256cce14bb3a2eb87d3f12fa164, State: Initialized, Role: LEARNER
I20260811 18:03:31.834111 3010 consensus_queue.cc:260] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: NON_VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: true } }
I20260811 18:03:31.837200 3010 ts_tablet_manager.cc:1428] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260811 18:03:31.839224 2409 tablet_copy_service.cc:342] P 17a4d3b2ce0f4459843cfaa371a5f03c: Request end of tablet copy session f57c0256cce14bb3a2eb87d3f12fa164-6525f6df4a774a8dba636d7b43d9f3f2 received from {username='slave'} at 127.0.0.1:48816
I20260811 18:03:31.839795 2409 tablet_copy_service.cc:433] P 17a4d3b2ce0f4459843cfaa371a5f03c: ending tablet copy session f57c0256cce14bb3a2eb87d3f12fa164-6525f6df4a774a8dba636d7b43d9f3f2 on tablet 6525f6df4a774a8dba636d7b43d9f3f2 with peer f57c0256cce14bb3a2eb87d3f12fa164
I20260811 18:03:32.034255 2377 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 18:03:32.036388 2498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:32.040526 2266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:32.040938 2613 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:32.306480 2278 raft_consensus.cc:1203] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[]
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
f5e200813e36468093d3fab3b5731c94 | 127.27.94.254:36017 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
17a4d3b2ce0f4459843cfaa371a5f03c | 127.27.94.194:45023 | HEALTHY | <none> | 1 | 0
962bcd545c534f88a3770241aa87c3cf | 127.27.94.195:34313 | HEALTHY | <none> | 0 | 0
adcc9b7c422d4ee28ba4acd7a32c634b | 127.27.94.196:33743 | HEALTHY | <none> | 0 | 0
f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 6525f6df4a774a8dba636d7b43d9f3f2 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
17a4d3b2ce0f4459843cfaa371a5f03c (127.27.94.194:45023): RUNNING [LEADER]
962bcd545c534f88a3770241aa87c3cf (127.27.94.195:34313): RUNNING
f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193:45985): RUNNING [NONVOTER]
All reported replicas are:
A = 17a4d3b2ce0f4459843cfaa371a5f03c
B = 962bcd545c534f88a3770241aa87c3cf
C = f57c0256cce14bb3a2eb87d3f12fa164
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 2 | 4 | Yes
B | A* B C~ | 2 | 4 | Yes
C | A B C~ | 2 | 4 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | adcc9b7c422d4ee28ba4acd7a32c634b | 127.27.94.196:33743 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260811 18:03:32.327350 28027 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260811 18:03:32.687618 3059 raft_consensus.cc:1050] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c: attempting to promote NON_VOTER f57c0256cce14bb3a2eb87d3f12fa164 to VOTER
I20260811 18:03:32.689450 3059 consensus_queue.cc:237] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: false } }
I20260811 18:03:32.695276 2278 raft_consensus.cc:1261] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164 [term 2 LEARNER]: Refusing update from remote peer 17a4d3b2ce0f4459843cfaa371a5f03c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260811 18:03:32.695367 2514 raft_consensus.cc:1261] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 2 FOLLOWER]: Refusing update from remote peer 17a4d3b2ce0f4459843cfaa371a5f03c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260811 18:03:32.697947 2982 consensus_queue.cc:1035] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260811 18:03:32.699240 3060 consensus_queue.cc:1035] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260811 18:03:32.709507 2982 raft_consensus.cc:2932] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: false } } }
I20260811 18:03:32.711481 2278 raft_consensus.cc:2932] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: false } } }
I20260811 18:03:32.712453 2514 raft_consensus.cc:2932] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: false } } }
I20260811 18:03:32.726090 2109 catalog_manager.cc:5495] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c reported cstate change: config changed from index 4 to 5, f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "17a4d3b2ce0f4459843cfaa371a5f03c" member_type: VOTER last_known_addr { host: "127.27.94.194" port: 45023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962bcd545c534f88a3770241aa87c3cf" member_type: VOTER last_known_addr { host: "127.27.94.195" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f57c0256cce14bb3a2eb87d3f12fa164" member_type: VOTER last_known_addr { host: "127.27.94.193" port: 45985 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260811 18:03:33.277300 2377 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260811 18:03:33.281493 2498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:33.281172 2613 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260811 18:03:33.293431 2266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
f5e200813e36468093d3fab3b5731c94 | 127.27.94.254:36017 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
17a4d3b2ce0f4459843cfaa371a5f03c | 127.27.94.194:45023 | HEALTHY | <none> | 1 | 0
962bcd545c534f88a3770241aa87c3cf | 127.27.94.195:34313 | HEALTHY | <none> | 0 | 0
adcc9b7c422d4ee28ba4acd7a32c634b | 127.27.94.196:33743 | HEALTHY | <none> | 0 | 0
f57c0256cce14bb3a2eb87d3f12fa164 | 127.27.94.193:45985 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | adcc9b7c422d4ee28ba4acd7a32c634b | 127.27.94.196:33743 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260811 18:03:33.698364 28027 log_verifier.cc:126] Checking tablet 6525f6df4a774a8dba636d7b43d9f3f2
I20260811 18:03:33.825067 28027 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 6525f6df4a774a8dba636d7b43d9f3f2
I20260811 18:03:33.831240 28027 tablet_server.cc:178] TabletServer@127.27.94.193:0 shutting down...
I20260811 18:03:33.898051 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:33.899228 28027 tablet_replica.cc:330] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164: stopping tablet replica
I20260811 18:03:33.900321 28027 raft_consensus.cc:2229] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:33.901060 28027 raft_consensus.cc:2258] T 6525f6df4a774a8dba636d7b43d9f3f2 P f57c0256cce14bb3a2eb87d3f12fa164 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:33.930737 28027 tablet_server.cc:194] TabletServer@127.27.94.193:0 shutdown complete.
I20260811 18:03:33.950456 28027 tablet_server.cc:178] TabletServer@127.27.94.194:0 shutting down...
W20260811 18:03:33.968632 2339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.94.193:45985: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20260811 18:03:33.978017 2339 consensus_peers.cc:487] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c -> Peer f57c0256cce14bb3a2eb87d3f12fa164 (127.27.94.193:45985): Couldn't send request to peer f57c0256cce14bb3a2eb87d3f12fa164. Status: Network error: Client connection negotiation failed: client connection to 127.27.94.193:45985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 18:03:34.017776 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:34.019004 28027 tablet_replica.cc:330] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c: stopping tablet replica
I20260811 18:03:34.020005 28027 raft_consensus.cc:2229] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 2 LEADER]: Raft consensus shutting down.
I20260811 18:03:34.021324 28027 raft_consensus.cc:2258] T 6525f6df4a774a8dba636d7b43d9f3f2 P 17a4d3b2ce0f4459843cfaa371a5f03c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:34.050843 28027 tablet_server.cc:194] TabletServer@127.27.94.194:0 shutdown complete.
I20260811 18:03:34.071535 28027 tablet_server.cc:178] TabletServer@127.27.94.195:0 shutting down...
I20260811 18:03:34.136711 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:34.138446 28027 tablet_replica.cc:330] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf: stopping tablet replica
I20260811 18:03:34.140324 28027 raft_consensus.cc:2229] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 2 FOLLOWER]: Raft consensus shutting down.
I20260811 18:03:34.141395 28027 raft_consensus.cc:2258] T 6525f6df4a774a8dba636d7b43d9f3f2 P 962bcd545c534f88a3770241aa87c3cf [term 2 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:34.173657 28027 tablet_server.cc:194] TabletServer@127.27.94.195:0 shutdown complete.
I20260811 18:03:34.194722 28027 tablet_server.cc:178] TabletServer@127.27.94.196:0 shutting down...
I20260811 18:03:34.258687 28027 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260811 18:03:34.283771 28027 tablet_server.cc:194] TabletServer@127.27.94.196:0 shutdown complete.
I20260811 18:03:34.308643 28027 master.cc:537] Master@127.27.94.254:36017 shutting down...
I20260811 18:03:34.356307 28027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 1 LEADER]: Raft consensus shutting down.
I20260811 18:03:34.357101 28027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 18:03:34.357734 28027 tablet_replica.cc:330] T 00000000000000000000000000000000 P f5e200813e36468093d3fab3b5731c94: stopping tablet replica
I20260811 18:03:34.386605 28027 master.cc:552] Master@127.27.94.254:36017 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (7936 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (12664 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (75471 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20260811 18:03:34.466764 28027 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~13 seconds ago
I20260811 18:03:34.467264 28027 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tserver/heartbeater.cc:644: suppressed but not reported on 54 messages since previous log ~64 seconds ago
I20260811 18:03:34.467525 28027 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tablet/tablet.cc:2369: suppressed but not reported on 8 messages since previous log ~17 seconds ago