[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 18:11:22.429482 822 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 1.097s user 0.459s sys 0.638s
W20260925 18:11:22.429962 822 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 1.098s user 0.459s sys 0.638s
I20260925 18:11:22.446318 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:46001
I20260925 18:11:22.447834 822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 18:11:22.449096 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:22.466078 834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:22.467027 833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:22.547796 836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:22.547806 822 server_base.cc:1034] running on GCE node
I20260925 18:11:22.549037 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:22.549249 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:22.549431 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359882549410 us; error 0 us; skew 500 ppm
I20260925 18:11:22.550149 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:22.558442 822 webserver.cc:460] Webserver started at http://127.0.205.190:39413/ using document root <none> and password file <none>
I20260925 18:11:22.559218 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:22.559407 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:22.559808 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:22.565573 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "0b5a69c606454a039687eae07a94986d"
format_stamp: "Formatted at 2026-09-25 18:11:22 on dist-test-slave-57m3"
I20260925 18:11:22.572592 822 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 18:11:22.577565 841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:22.578653 822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 18:11:22.578971 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "0b5a69c606454a039687eae07a94986d"
format_stamp: "Formatted at 2026-09-25 18:11:22 on dist-test-slave-57m3"
I20260925 18:11:22.579284 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:22.603256 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:22.604662 822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 18:11:22.605113 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:22.678659 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:46001
I20260925 18:11:22.678732 892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:46001 every 8 connection(s)
I20260925 18:11:22.684978 893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:22.708568 893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d: Bootstrap starting.
I20260925 18:11:22.714600 893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:22.716269 893 log.cc:824] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d: Log is configured to *not* fsync() on all Append() calls
I20260925 18:11:22.720791 893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d: No bootstrap required, opened a new log
I20260925 18:11:22.743552 893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5a69c606454a039687eae07a94986d" member_type: VOTER }
I20260925 18:11:22.744083 893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:22.744305 893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5a69c606454a039687eae07a94986d, State: Initialized, Role: FOLLOWER
I20260925 18:11:22.745083 893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5a69c606454a039687eae07a94986d" member_type: VOTER }
I20260925 18:11:22.745553 893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:11:22.745776 893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:11:22.746034 893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:22.749764 893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5a69c606454a039687eae07a94986d" member_type: VOTER }
I20260925 18:11:22.750422 893 leader_election.cc:302] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b5a69c606454a039687eae07a94986d; no voters:
I20260925 18:11:22.752310 893 leader_election.cc:288] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:11:22.752655 896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:11:22.755138 896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 1 LEADER]: Becoming Leader. State: Replica: 0b5a69c606454a039687eae07a94986d, State: Running, Role: LEADER
I20260925 18:11:22.755888 896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5a69c606454a039687eae07a94986d" member_type: VOTER }
I20260925 18:11:22.756934 893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:11:22.764815 897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b5a69c606454a039687eae07a94986d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5a69c606454a039687eae07a94986d" member_type: VOTER } }
I20260925 18:11:22.764981 898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b5a69c606454a039687eae07a94986d. Latest consensus state: current_term: 1 leader_uuid: "0b5a69c606454a039687eae07a94986d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5a69c606454a039687eae07a94986d" member_type: VOTER } }
I20260925 18:11:22.765612 897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:22.765623 898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:22.773334 904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:11:22.784061 904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:11:22.788566 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:11:22.801456 904 catalog_manager.cc:1347] Generated new cluster ID: 44367e7976b8463ebdf21d4df959e0ab
I20260925 18:11:22.801741 904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:11:22.816753 904 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:11:22.818593 904 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:11:22.832168 904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d: Generated new TSK 0
I20260925 18:11:22.833204 904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:11:22.858474 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:22.867861 915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:22.870846 916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:22.873466 918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:22.874586 822 server_base.cc:1034] running on GCE node
I20260925 18:11:22.875494 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:22.875764 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:22.876005 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359882875980 us; error 0 us; skew 500 ppm
I20260925 18:11:22.876760 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:22.880877 822 webserver.cc:460] Webserver started at http://127.0.205.129:35367/ using document root <none> and password file <none>
I20260925 18:11:22.881565 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:22.881822 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:22.882143 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:22.883817 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "fc94d8e967ec44bba783aeb4c05357c0"
format_stamp: "Formatted at 2026-09-25 18:11:22 on dist-test-slave-57m3"
I20260925 18:11:22.891235 822 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260925 18:11:22.896324 923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:22.897177 822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 18:11:22.897516 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "fc94d8e967ec44bba783aeb4c05357c0"
format_stamp: "Formatted at 2026-09-25 18:11:22 on dist-test-slave-57m3"
I20260925 18:11:22.897984 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:22.928555 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:22.930219 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:22.933310 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:22.938899 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:22.939198 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:22.939489 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:22.939697 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:23.041021 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:40781
I20260925 18:11:23.041126 985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:40781 every 8 connection(s)
I20260925 18:11:23.047760 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:23.066983 990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:23.070309 991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:23.081594 993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:23.087318 822 server_base.cc:1034] running on GCE node
I20260925 18:11:23.088083 986 heartbeater.cc:347] Connected to a master server at 127.0.205.190:46001
I20260925 18:11:23.088218 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:23.088449 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:23.088532 986 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:23.088652 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359883088633 us; error 0 us; skew 500 ppm
I20260925 18:11:23.089347 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:23.089627 986 heartbeater.cc:511] Master 127.0.205.190:46001 requested a full tablet report, sending...
I20260925 18:11:23.093135 858 ts_manager.cc:194] Registered new tserver with Master: fc94d8e967ec44bba783aeb4c05357c0 (127.0.205.129:40781)
I20260925 18:11:23.093408 822 webserver.cc:460] Webserver started at http://127.0.205.130:33767/ using document root <none> and password file <none>
I20260925 18:11:23.093895 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:23.094071 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:23.094324 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:23.095324 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/ts-1-root/instance:
uuid: "db5edafc43ff42048864ff006f8f474b"
format_stamp: "Formatted at 2026-09-25 18:11:23 on dist-test-slave-57m3"
I20260925 18:11:23.096105 858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37124
I20260925 18:11:23.101451 822 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260925 18:11:23.105685 998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:23.106456 822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 18:11:23.106709 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/ts-1-root
uuid: "db5edafc43ff42048864ff006f8f474b"
format_stamp: "Formatted at 2026-09-25 18:11:23 on dist-test-slave-57m3"
I20260925 18:11:23.106988 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1790359881141916-822-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:23.129285 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:23.130415 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:23.132804 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:23.135563 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:23.135748 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:23.135984 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:23.136126 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:23.200578 986 heartbeater.cc:503] Master 127.0.205.190:46001 was elected leader, sending a full tablet report...
I20260925 18:11:23.229840 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.130:35585
I20260925 18:11:23.229916 1060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.130:35585 every 8 connection(s)
I20260925 18:11:23.251331 1061 heartbeater.cc:347] Connected to a master server at 127.0.205.190:46001
I20260925 18:11:23.251717 1061 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:23.252568 1061 heartbeater.cc:511] Master 127.0.205.190:46001 requested a full tablet report, sending...
I20260925 18:11:23.255191 858 ts_manager.cc:194] Registered new tserver with Master: db5edafc43ff42048864ff006f8f474b (127.0.205.130:35585)
I20260925 18:11:23.256986 858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37134
I20260925 18:11:23.257014 822 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.019150543s
I20260925 18:11:23.288873 858 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20260925 18:11:23.315284 858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260925 18:11:23.360968 1061 heartbeater.cc:503] Master 127.0.205.190:46001 was elected leader, sending a full tablet report...
I20260925 18:11:23.369360 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 71335311fa4c460e8c30fc4090cadf27 (DEFAULT_TABLE table=test-table [id=a3c6f46808f240f9b092e3590e93c921]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:23.369360 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc9a963c4f74b0093f039a6f0d29d92 (DEFAULT_TABLE table=test-table [id=a3c6f46808f240f9b092e3590e93c921]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:23.369889 950 tablet_service.cc:1467] Processing CreateTablet for tablet 71335311fa4c460e8c30fc4090cadf27 (DEFAULT_TABLE table=test-table [id=a3c6f46808f240f9b092e3590e93c921]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:23.372161 950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71335311fa4c460e8c30fc4090cadf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:23.374217 951 tablet_service.cc:1467] Processing CreateTablet for tablet 6bc9a963c4f74b0093f039a6f0d29d92 (DEFAULT_TABLE table=test-table [id=a3c6f46808f240f9b092e3590e93c921]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:23.377442 951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc9a963c4f74b0093f039a6f0d29d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:23.379483 1025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71335311fa4c460e8c30fc4090cadf27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:23.380537 1026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bc9a963c4f74b0093f039a6f0d29d92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:23.405395 1075 tablet_bootstrap.cc:492] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b: Bootstrap starting.
I20260925 18:11:23.410679 1074 tablet_bootstrap.cc:492] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0: Bootstrap starting.
I20260925 18:11:23.414259 1075 tablet_bootstrap.cc:654] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:23.416218 1074 tablet_bootstrap.cc:654] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:23.421272 1075 tablet_bootstrap.cc:492] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b: No bootstrap required, opened a new log
I20260925 18:11:23.422004 1075 ts_tablet_manager.cc:1397] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20260925 18:11:23.424671 1075 raft_consensus.cc:348] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:23.425380 1074 tablet_bootstrap.cc:492] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0: No bootstrap required, opened a new log
I20260925 18:11:23.425412 1075 raft_consensus.cc:729] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5edafc43ff42048864ff006f8f474b, State: Initialized, Role: FOLLOWER
I20260925 18:11:23.425871 1074 ts_tablet_manager.cc:1397] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260925 18:11:23.426086 1075 consensus_queue.cc:260] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:23.428205 1074 raft_consensus.cc:348] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:23.428912 1074 raft_consensus.cc:729] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc94d8e967ec44bba783aeb4c05357c0, State: Initialized, Role: FOLLOWER
I20260925 18:11:23.429476 1074 consensus_queue.cc:260] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:23.429870 1075 ts_tablet_manager.cc:1428] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260925 18:11:23.431092 1075 tablet_bootstrap.cc:492] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b: Bootstrap starting.
I20260925 18:11:23.438537 1075 tablet_bootstrap.cc:654] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:23.442600 1074 ts_tablet_manager.cc:1428] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0: Time spent starting tablet: real 0.016s user 0.005s sys 0.008s
I20260925 18:11:23.443411 1074 tablet_bootstrap.cc:492] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0: Bootstrap starting.
I20260925 18:11:23.448992 1074 tablet_bootstrap.cc:654] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:23.454754 1075 tablet_bootstrap.cc:492] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b: No bootstrap required, opened a new log
I20260925 18:11:23.455113 1075 ts_tablet_manager.cc:1397] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.004s
I20260925 18:11:23.457378 1075 raft_consensus.cc:348] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } } peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } }
I20260925 18:11:23.457829 1075 raft_consensus.cc:729] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5edafc43ff42048864ff006f8f474b, State: Initialized, Role: FOLLOWER
I20260925 18:11:23.458297 1075 consensus_queue.cc:260] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } } peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } }
I20260925 18:11:23.459842 1075 ts_tablet_manager.cc:1428] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 18:11:23.465636 1074 tablet_bootstrap.cc:492] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0: No bootstrap required, opened a new log
I20260925 18:11:23.465976 1074 ts_tablet_manager.cc:1397] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.007s
I20260925 18:11:23.468950 1074 raft_consensus.cc:348] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } } peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } }
I20260925 18:11:23.469539 1074 raft_consensus.cc:729] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc94d8e967ec44bba783aeb4c05357c0, State: Initialized, Role: FOLLOWER
I20260925 18:11:23.470044 1074 consensus_queue.cc:260] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } } peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } }
I20260925 18:11:23.471930 1074 ts_tablet_manager.cc:1428] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260925 18:11:23.488783 1062 tablet.cc:2369] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 18:11:23.511130 822 ts_tablet_manager-itest.cc:367] Electing peer 2...
I20260925 18:11:23.511448 822 raft_consensus.cc:412] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 0 FOLLOWER]: Emulating election...
I20260925 18:11:23.511704 822 raft_consensus.cc:3037] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:23.517732 822 raft_consensus.cc:686] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 1 LEADER]: Becoming Leader. State: Replica: db5edafc43ff42048864ff006f8f474b, State: Running, Role: LEADER
I20260925 18:11:23.518409 822 consensus_queue.cc:237] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } } peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } }
I20260925 18:11:23.525056 822 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260925 18:11:23.528736 857 catalog_manager.cc:5495] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b reported cstate change: term changed from 0 to 1, leader changed from <none> to db5edafc43ff42048864ff006f8f474b (127.0.205.130). New cstate: current_term: 1 leader_uuid: "db5edafc43ff42048864ff006f8f474b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } health_report { overall_health: HEALTHY } } }
I20260925 18:11:23.553300 822 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260925 18:11:23.657533 822 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260925 18:11:23.861123 822 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260925 18:11:24.059150 961 raft_consensus.cc:3037] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:24.068228 1085 consensus_queue.cc:1035] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:24.164740 822 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260925 18:11:24.165083 822 raft_consensus.cc:412] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 0 FOLLOWER]: Emulating election...
I20260925 18:11:24.165361 822 raft_consensus.cc:3037] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:24.168828 822 raft_consensus.cc:686] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 1 LEADER]: Becoming Leader. State: Replica: db5edafc43ff42048864ff006f8f474b, State: Running, Role: LEADER
I20260925 18:11:24.169443 822 consensus_queue.cc:237] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:24.171571 822 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260925 18:11:24.175513 822 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260925 18:11:24.175594 857 catalog_manager.cc:5495] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b reported cstate change: term changed from 0 to 1, leader changed from <none> to db5edafc43ff42048864ff006f8f474b (127.0.205.130). New cstate: current_term: 1 leader_uuid: "db5edafc43ff42048864ff006f8f474b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } health_report { overall_health: UNKNOWN } } }
I20260925 18:11:24.279345 822 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260925 18:11:24.483114 822 cluster_itest_util.cc:256] Not converged past 0 yet: 0.0 1.1
I20260925 18:11:24.694295 961 raft_consensus.cc:3037] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:24.700287 1090 consensus_queue.cc:1035] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:24.986901 822 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260925 18:11:24.987188 822 raft_consensus.cc:412] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 1 LEADER]: Emulating election...
I20260925 18:11:24.987447 822 raft_consensus.cc:3032] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 1 LEADER]: Stepping down as leader of term 1
I20260925 18:11:24.987689 822 raft_consensus.cc:729] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: db5edafc43ff42048864ff006f8f474b, State: Running, Role: LEADER
I20260925 18:11:24.988240 822 consensus_queue.cc:260] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:24.989148 822 raft_consensus.cc:3037] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:24.992610 822 raft_consensus.cc:686] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 2 LEADER]: Becoming Leader. State: Replica: db5edafc43ff42048864ff006f8f474b, State: Running, Role: LEADER
I20260925 18:11:24.993181 822 consensus_queue.cc:237] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:24.994649 822 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260925 18:11:24.997694 858 catalog_manager.cc:5495] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "db5edafc43ff42048864ff006f8f474b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } health_report { overall_health: UNKNOWN } } }
I20260925 18:11:24.998065 822 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260925 18:11:25.101620 822 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260925 18:11:25.305194 822 cluster_itest_util.cc:256] Not converged past 0 yet: 1.1 2.2
I20260925 18:11:25.388751 961 raft_consensus.cc:3037] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:25.393179 961 raft_consensus.cc:1261] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 2 FOLLOWER]: Refusing update from remote peer db5edafc43ff42048864ff006f8f474b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 18:11:25.394327 1090 consensus_queue.cc:1035] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 18:11:25.608624 822 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260925 18:11:25.608861 822 raft_consensus.cc:412] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [term 1 FOLLOWER]: Emulating election...
I20260925 18:11:25.609067 822 raft_consensus.cc:3037] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:25.614006 822 raft_consensus.cc:686] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [term 2 LEADER]: Becoming Leader. State: Replica: fc94d8e967ec44bba783aeb4c05357c0, State: Running, Role: LEADER
I20260925 18:11:25.614637 822 consensus_queue.cc:237] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } } peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } }
I20260925 18:11:25.616588 822 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260925 18:11:25.619793 822 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260925 18:11:25.619997 858 catalog_manager.cc:5495] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 reported cstate change: term changed from 1 to 2, leader changed from db5edafc43ff42048864ff006f8f474b (127.0.205.130) to fc94d8e967ec44bba783aeb4c05357c0 (127.0.205.129). New cstate: current_term: 2 leader_uuid: "fc94d8e967ec44bba783aeb4c05357c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } health_report { overall_health: UNKNOWN } } }
I20260925 18:11:25.654687 961 raft_consensus.cc:1226] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [term 2 LEADER]: Rejecting Update request from peer db5edafc43ff42048864ff006f8f474b for earlier term 1. Current term is 2. Ops: []
I20260925 18:11:25.655652 1106 consensus_queue.cc:1046] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 18:11:25.656212 1106 consensus_queue.cc:1225] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [LEADER]: Peer fc94d8e967ec44bba783aeb4c05357c0 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260925 18:11:25.656560 1090 raft_consensus.cc:3032] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 1 LEADER]: Stepping down as leader of term 1
I20260925 18:11:25.656823 1090 raft_consensus.cc:729] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: db5edafc43ff42048864ff006f8f474b, State: Running, Role: LEADER
I20260925 18:11:25.657394 1090 consensus_queue.cc:260] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } } peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } }
I20260925 18:11:25.658097 1090 raft_consensus.cc:3037] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:25.723380 822 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260925 18:11:25.927153 822 cluster_itest_util.cc:256] Not converged past 0 yet: 2.2 1.1
I20260925 18:11:26.044095 1036 raft_consensus.cc:1261] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 2 FOLLOWER]: Refusing update from remote peer fc94d8e967ec44bba783aeb4c05357c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 18:11:26.045240 1105 consensus_queue.cc:1035] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 18:11:26.430871 822 ts_tablet_manager-itest.cc:367] Electing peer 1...
I20260925 18:11:26.431181 822 raft_consensus.cc:412] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 2 FOLLOWER]: Emulating election...
I20260925 18:11:26.431468 822 raft_consensus.cc:3037] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:26.435341 822 raft_consensus.cc:686] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 3 LEADER]: Becoming Leader. State: Replica: fc94d8e967ec44bba783aeb4c05357c0, State: Running, Role: LEADER
I20260925 18:11:26.435822 822 consensus_queue.cc:237] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:26.437525 822 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260925 18:11:26.441170 822 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260925 18:11:26.441087 856 catalog_manager.cc:5495] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 reported cstate change: term changed from 2 to 3, leader changed from db5edafc43ff42048864ff006f8f474b (127.0.205.130) to fc94d8e967ec44bba783aeb4c05357c0 (127.0.205.129). New cstate: current_term: 3 leader_uuid: "fc94d8e967ec44bba783aeb4c05357c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } health_report { overall_health: HEALTHY } } }
I20260925 18:11:26.545394 822 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260925 18:11:26.748764 822 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 2.2
I20260925 18:11:26.844525 961 raft_consensus.cc:1226] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 3 LEADER]: Rejecting Update request from peer db5edafc43ff42048864ff006f8f474b for earlier term 2. Current term is 3. Ops: []
I20260925 18:11:26.845548 1123 consensus_queue.cc:1046] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:11:26.846683 1123 consensus_queue.cc:1225] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [LEADER]: Peer fc94d8e967ec44bba783aeb4c05357c0 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260925 18:11:26.847138 1124 raft_consensus.cc:3032] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 2 LEADER]: Stepping down as leader of term 2
I20260925 18:11:26.847384 1124 raft_consensus.cc:729] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: db5edafc43ff42048864ff006f8f474b, State: Running, Role: LEADER
I20260925 18:11:26.847846 1124 consensus_queue.cc:260] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:26.848495 1124 raft_consensus.cc:3037] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:27.042632 1036 raft_consensus.cc:1261] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 3 FOLLOWER]: Refusing update from remote peer fc94d8e967ec44bba783aeb4c05357c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 18:11:27.043782 1125 consensus_queue.cc:1035] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:11:27.057857 822 ts_tablet_manager-itest.cc:367] Electing peer 3...
I20260925 18:11:27.058161 822 raft_consensus.cc:412] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 3 FOLLOWER]: Emulating election...
I20260925 18:11:27.058386 822 raft_consensus.cc:3037] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:11:27.063215 822 raft_consensus.cc:686] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 4 LEADER]: Becoming Leader. State: Replica: db5edafc43ff42048864ff006f8f474b, State: Running, Role: LEADER
I20260925 18:11:27.063845 822 consensus_queue.cc:237] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:27.065356 822 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260925 18:11:27.068506 822 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260925 18:11:27.068602 856 catalog_manager.cc:5495] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b reported cstate change: term changed from 3 to 4, leader changed from fc94d8e967ec44bba783aeb4c05357c0 (127.0.205.129) to db5edafc43ff42048864ff006f8f474b (127.0.205.130). New cstate: current_term: 4 leader_uuid: "db5edafc43ff42048864ff006f8f474b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } health_report { overall_health: UNKNOWN } } }
I20260925 18:11:27.172331 822 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260925 18:11:27.375631 822 cluster_itest_util.cc:256] Not converged past 0 yet: 3.3 4.4
I20260925 18:11:27.504915 1036 raft_consensus.cc:1226] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 4 LEADER]: Rejecting Update request from peer fc94d8e967ec44bba783aeb4c05357c0 for earlier term 3. Current term is 4. Ops: []
I20260925 18:11:27.505991 1125 consensus_queue.cc:1046] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 18:11:27.506505 1125 consensus_queue.cc:1225] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [LEADER]: Peer db5edafc43ff42048864ff006f8f474b log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260925 18:11:27.506858 1127 raft_consensus.cc:3032] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 18:11:27.507073 1127 raft_consensus.cc:729] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fc94d8e967ec44bba783aeb4c05357c0, State: Running, Role: LEADER
I20260925 18:11:27.507474 1127 consensus_queue.cc:260] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5edafc43ff42048864ff006f8f474b" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35585 } } peers { permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 } }
I20260925 18:11:27.508566 1127 raft_consensus.cc:3037] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:11:27.619724 961 raft_consensus.cc:1261] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 4 FOLLOWER]: Refusing update from remote peer db5edafc43ff42048864ff006f8f474b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 18:11:27.620966 1133 consensus_queue.cc:1035] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc94d8e967ec44bba783aeb4c05357c0" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 18:11:27.880970 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:11:27.898443 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:27.898998 822 tablet_replica.cc:330] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0: stopping tablet replica
I20260925 18:11:27.899502 822 raft_consensus.cc:2229] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [term 2 LEADER]: Raft consensus shutting down.
I20260925 18:11:27.900094 822 raft_consensus.cc:2258] T 71335311fa4c460e8c30fc4090cadf27 P fc94d8e967ec44bba783aeb4c05357c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:27.901889 822 tablet_replica.cc:330] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0: stopping tablet replica
I20260925 18:11:27.902256 822 raft_consensus.cc:2229] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260925 18:11:27.902554 822 raft_consensus.cc:2258] T 6bc9a963c4f74b0093f039a6f0d29d92 P fc94d8e967ec44bba783aeb4c05357c0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:27.981233 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:11:27.989984 822 tablet_server.cc:178] TabletServer@127.0.205.130:0 shutting down...
I20260925 18:11:28.005173 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:28.005833 822 tablet_replica.cc:330] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b: stopping tablet replica
I20260925 18:11:28.006422 822 raft_consensus.cc:2229] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 18:11:28.006901 822 raft_consensus.cc:2258] T 71335311fa4c460e8c30fc4090cadf27 P db5edafc43ff42048864ff006f8f474b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:28.008872 822 tablet_replica.cc:330] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b: stopping tablet replica
I20260925 18:11:28.009431 822 raft_consensus.cc:2229] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 4 LEADER]: Raft consensus shutting down.
I20260925 18:11:28.010103 822 raft_consensus.cc:2258] T 6bc9a963c4f74b0093f039a6f0d29d92 P db5edafc43ff42048864ff006f8f474b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:28.030182 822 tablet_server.cc:194] TabletServer@127.0.205.130:0 shutdown complete.
I20260925 18:11:28.037568 822 master.cc:537] Master@127.0.205.190:46001 shutting down...
I20260925 18:11:28.199678 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:11:28.200140 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:28.200423 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0b5a69c606454a039687eae07a94986d: stopping tablet replica
I20260925 18:11:28.216663 822 master.cc:552] Master@127.0.205.190:46001 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (6937 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20260925 18:11:28.241474 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:37317
I20260925 18:11:28.242450 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:28.246791 1141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:28.247880 1142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:28.249294 1144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:28.250253 822 server_base.cc:1034] running on GCE node
I20260925 18:11:28.250816 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:28.250970 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:28.251122 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359888251109 us; error 0 us; skew 500 ppm
I20260925 18:11:28.251605 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:28.257272 822 webserver.cc:460] Webserver started at http://127.0.205.190:42949/ using document root <none> and password file <none>
I20260925 18:11:28.257714 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:28.257869 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:28.258106 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:28.259064 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "dec5353be8bd48b79b72204b75fd1d22"
format_stamp: "Formatted at 2026-09-25 18:11:28 on dist-test-slave-57m3"
I20260925 18:11:28.263214 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 18:11:28.266072 1149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:28.266711 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 18:11:28.266944 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "dec5353be8bd48b79b72204b75fd1d22"
format_stamp: "Formatted at 2026-09-25 18:11:28 on dist-test-slave-57m3"
I20260925 18:11:28.267185 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:28.282781 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:28.283705 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:28.314518 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:37317
I20260925 18:11:28.314612 1200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:37317 every 8 connection(s)
I20260925 18:11:28.317869 1201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:28.326072 1201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22: Bootstrap starting.
I20260925 18:11:28.329612 1201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:28.332930 1201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22: No bootstrap required, opened a new log
I20260925 18:11:28.334517 1201 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec5353be8bd48b79b72204b75fd1d22" member_type: VOTER }
I20260925 18:11:28.334856 1201 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:28.335070 1201 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec5353be8bd48b79b72204b75fd1d22, State: Initialized, Role: FOLLOWER
I20260925 18:11:28.335539 1201 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec5353be8bd48b79b72204b75fd1d22" member_type: VOTER }
I20260925 18:11:28.335935 1201 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:11:28.336136 1201 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:11:28.336366 1201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:28.340011 1201 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec5353be8bd48b79b72204b75fd1d22" member_type: VOTER }
I20260925 18:11:28.340452 1201 leader_election.cc:302] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec5353be8bd48b79b72204b75fd1d22; no voters:
I20260925 18:11:28.341521 1201 leader_election.cc:288] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:11:28.341840 1204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:11:28.343024 1204 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 1 LEADER]: Becoming Leader. State: Replica: dec5353be8bd48b79b72204b75fd1d22, State: Running, Role: LEADER
I20260925 18:11:28.343518 1204 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec5353be8bd48b79b72204b75fd1d22" member_type: VOTER }
I20260925 18:11:28.344079 1201 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:11:28.346091 1206 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec5353be8bd48b79b72204b75fd1d22. Latest consensus state: current_term: 1 leader_uuid: "dec5353be8bd48b79b72204b75fd1d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec5353be8bd48b79b72204b75fd1d22" member_type: VOTER } }
I20260925 18:11:28.346143 1205 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec5353be8bd48b79b72204b75fd1d22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec5353be8bd48b79b72204b75fd1d22" member_type: VOTER } }
I20260925 18:11:28.346647 1206 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:28.346743 1205 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:28.349642 1209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:11:28.354096 1209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:11:28.357326 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:11:28.361392 1209 catalog_manager.cc:1347] Generated new cluster ID: 3f41fcffeb0a4b52bbf1a09adcb3af10
I20260925 18:11:28.361601 1209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:11:28.383456 1209 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:11:28.385143 1209 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:11:28.396147 1209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22: Generated new TSK 0
I20260925 18:11:28.396690 1209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:11:28.423102 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:28.428494 1222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:28.429570 1223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:28.430461 1225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:28.431249 822 server_base.cc:1034] running on GCE node
I20260925 18:11:28.431990 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:28.432152 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:28.432261 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359888432253 us; error 0 us; skew 500 ppm
I20260925 18:11:28.432672 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:28.434784 822 webserver.cc:460] Webserver started at http://127.0.205.129:32981/ using document root <none> and password file <none>
I20260925 18:11:28.435178 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:28.435326 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:28.435527 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:28.436462 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "1ca163d2fc9f47f1acc0220852bb7827"
format_stamp: "Formatted at 2026-09-25 18:11:28 on dist-test-slave-57m3"
I20260925 18:11:28.440457 822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 18:11:28.443280 1230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:28.443924 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:11:28.444231 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "1ca163d2fc9f47f1acc0220852bb7827"
format_stamp: "Formatted at 2026-09-25 18:11:28 on dist-test-slave-57m3"
I20260925 18:11:28.444515 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:28.458418 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:28.459350 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:28.460570 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:28.462639 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:28.462819 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:28.463044 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:28.463181 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:28.496322 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:42675
I20260925 18:11:28.496420 1292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:42675 every 8 connection(s)
I20260925 18:11:28.500259 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:28.506737 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:28.508764 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:28.510058 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:28.510867 822 server_base.cc:1034] running on GCE node
I20260925 18:11:28.511667 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:28.511853 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:28.512032 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359888512018 us; error 0 us; skew 500 ppm
I20260925 18:11:28.512691 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:28.514389 1293 heartbeater.cc:347] Connected to a master server at 127.0.205.190:37317
I20260925 18:11:28.514676 1293 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:28.515319 1293 heartbeater.cc:511] Master 127.0.205.190:37317 requested a full tablet report, sending...
I20260925 18:11:28.515473 822 webserver.cc:460] Webserver started at http://127.0.205.130:40505/ using document root <none> and password file <none>
I20260925 18:11:28.516011 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:28.516252 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:28.516557 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:28.517253 1166 ts_manager.cc:194] Registered new tserver with Master: 1ca163d2fc9f47f1acc0220852bb7827 (127.0.205.129:42675)
I20260925 18:11:28.517706 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/ts-1-root/instance:
uuid: "3219d9d691604d5b8b5456211d0a5c96"
format_stamp: "Formatted at 2026-09-25 18:11:28 on dist-test-slave-57m3"
I20260925 18:11:28.518934 1166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59818
I20260925 18:11:28.522251 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 18:11:28.525359 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:28.526050 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260925 18:11:28.526304 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/ts-1-root
uuid: "3219d9d691604d5b8b5456211d0a5c96"
format_stamp: "Formatted at 2026-09-25 18:11:28 on dist-test-slave-57m3"
I20260925 18:11:28.526578 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1790359881141916-822-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:28.539448 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:28.540441 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:28.541764 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:28.543745 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:28.543969 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:28.544185 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:28.544322 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:28.578191 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.130:38225
I20260925 18:11:28.578271 1367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.130:38225 every 8 connection(s)
I20260925 18:11:28.592397 1368 heartbeater.cc:347] Connected to a master server at 127.0.205.190:37317
I20260925 18:11:28.592744 1368 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:28.593386 1368 heartbeater.cc:511] Master 127.0.205.190:37317 requested a full tablet report, sending...
I20260925 18:11:28.595078 1166 ts_manager.cc:194] Registered new tserver with Master: 3219d9d691604d5b8b5456211d0a5c96 (127.0.205.130:38225)
I20260925 18:11:28.595160 822 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.014230514s
I20260925 18:11:28.596458 1166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59826
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:421: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20260925 18:11:28.615680 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:11:28.621986 1293 heartbeater.cc:503] Master 127.0.205.190:37317 was elected leader, sending a full tablet report...
I20260925 18:11:28.635094 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:28.650174 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:11:28.656301 822 tablet_server.cc:178] TabletServer@127.0.205.130:0 shutting down...
I20260925 18:11:28.667933 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:28.682713 822 tablet_server.cc:194] TabletServer@127.0.205.130:0 shutdown complete.
I20260925 18:11:28.689021 822 master.cc:537] Master@127.0.205.190:37317 shutting down...
I20260925 18:11:28.700409 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:11:28.700925 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:28.701295 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P dec5353be8bd48b79b72204b75fd1d22: stopping tablet replica
I20260925 18:11:28.718178 822 master.cc:552] Master@127.0.205.190:37317 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (497 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20260925 18:11:28.738044 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:37917
I20260925 18:11:28.738974 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:28.743937 1381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:28.744589 1382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:28.747390 822 server_base.cc:1034] running on GCE node
W20260925 18:11:28.747329 1384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:28.748152 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:28.748328 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:28.748476 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359888748463 us; error 0 us; skew 500 ppm
I20260925 18:11:28.749063 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:28.751184 822 webserver.cc:460] Webserver started at http://127.0.205.190:34095/ using document root <none> and password file <none>
I20260925 18:11:28.751607 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:28.751776 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:28.752019 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:28.753069 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "d36291bd9c624769b8f956b4fa169477"
format_stamp: "Formatted at 2026-09-25 18:11:28 on dist-test-slave-57m3"
I20260925 18:11:28.757120 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 18:11:28.760074 1389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:28.760749 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 18:11:28.761056 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "d36291bd9c624769b8f956b4fa169477"
format_stamp: "Formatted at 2026-09-25 18:11:28 on dist-test-slave-57m3"
I20260925 18:11:28.761315 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:28.778919 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:28.779913 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:28.812343 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:37917
I20260925 18:11:28.812449 1440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:37917 every 8 connection(s)
I20260925 18:11:28.815718 1441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:28.825103 1441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477: Bootstrap starting.
I20260925 18:11:28.829098 1441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:28.832644 1441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477: No bootstrap required, opened a new log
I20260925 18:11:28.834646 1441 raft_consensus.cc:348] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36291bd9c624769b8f956b4fa169477" member_type: VOTER }
I20260925 18:11:28.835026 1441 raft_consensus.cc:374] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:28.835201 1441 raft_consensus.cc:729] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36291bd9c624769b8f956b4fa169477, State: Initialized, Role: FOLLOWER
I20260925 18:11:28.835666 1441 consensus_queue.cc:260] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36291bd9c624769b8f956b4fa169477" member_type: VOTER }
I20260925 18:11:28.836059 1441 raft_consensus.cc:388] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:11:28.836309 1441 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:11:28.836544 1441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:28.840507 1441 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36291bd9c624769b8f956b4fa169477" member_type: VOTER }
I20260925 18:11:28.841045 1441 leader_election.cc:302] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d36291bd9c624769b8f956b4fa169477; no voters:
I20260925 18:11:28.842031 1441 leader_election.cc:288] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:11:28.842276 1444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:11:28.843425 1444 raft_consensus.cc:686] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 1 LEADER]: Becoming Leader. State: Replica: d36291bd9c624769b8f956b4fa169477, State: Running, Role: LEADER
I20260925 18:11:28.844018 1444 consensus_queue.cc:237] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36291bd9c624769b8f956b4fa169477" member_type: VOTER }
I20260925 18:11:28.844518 1441 sys_catalog.cc:564] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:11:28.846714 1446 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36291bd9c624769b8f956b4fa169477. Latest consensus state: current_term: 1 leader_uuid: "d36291bd9c624769b8f956b4fa169477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36291bd9c624769b8f956b4fa169477" member_type: VOTER } }
I20260925 18:11:28.846706 1445 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d36291bd9c624769b8f956b4fa169477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36291bd9c624769b8f956b4fa169477" member_type: VOTER } }
I20260925 18:11:28.847265 1446 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:28.847541 1445 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:28.854419 1449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:11:28.861145 1449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:11:28.862121 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:11:28.869170 1449 catalog_manager.cc:1347] Generated new cluster ID: bd5d84f906f34cb5a64802975c9d262d
I20260925 18:11:28.869442 1449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:11:28.884261 1449 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:11:28.885819 1449 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:11:28.896203 1449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477: Generated new TSK 0
I20260925 18:11:28.896857 1449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:11:28.928560 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:28.933758 1462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:28.934664 1463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:28.936610 822 server_base.cc:1034] running on GCE node
W20260925 18:11:28.936569 1465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:28.937520 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:28.937695 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:28.937834 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359888937821 us; error 0 us; skew 500 ppm
I20260925 18:11:28.938290 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:28.940305 822 webserver.cc:460] Webserver started at http://127.0.205.129:40605/ using document root <none> and password file <none>
I20260925 18:11:28.940757 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:28.940958 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:28.941198 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:28.942134 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "618e2faf69a645568e3c8d9cf46705a5"
format_stamp: "Formatted at 2026-09-25 18:11:28 on dist-test-slave-57m3"
I20260925 18:11:28.946652 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 18:11:28.949486 1470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:28.950141 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 18:11:28.950383 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "618e2faf69a645568e3c8d9cf46705a5"
format_stamp: "Formatted at 2026-09-25 18:11:28 on dist-test-slave-57m3"
I20260925 18:11:28.950649 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:28.970709 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:28.971688 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:28.972987 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:28.974992 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:28.975167 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:28.975371 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:28.975504 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:29.010924 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:37769
I20260925 18:11:29.010987 1532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:37769 every 8 connection(s)
I20260925 18:11:29.015271 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:29.021872 1537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:29.024416 1538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:29.025477 1540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:29.026072 822 server_base.cc:1034] running on GCE node
I20260925 18:11:29.026726 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:29.027014 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:29.027208 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359889027190 us; error 0 us; skew 500 ppm
I20260925 18:11:29.027818 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:29.028342 1533 heartbeater.cc:347] Connected to a master server at 127.0.205.190:37917
I20260925 18:11:29.028769 1533 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:29.029568 1533 heartbeater.cc:511] Master 127.0.205.190:37917 requested a full tablet report, sending...
I20260925 18:11:29.030457 822 webserver.cc:460] Webserver started at http://127.0.205.130:39785/ using document root <none> and password file <none>
I20260925 18:11:29.030900 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:29.031113 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:29.031396 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:29.031584 1406 ts_manager.cc:194] Registered new tserver with Master: 618e2faf69a645568e3c8d9cf46705a5 (127.0.205.129:37769)
I20260925 18:11:29.032778 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/ts-1-root/instance:
uuid: "0b51096c8830473ca018f49c58e3a613"
format_stamp: "Formatted at 2026-09-25 18:11:29 on dist-test-slave-57m3"
I20260925 18:11:29.034178 1406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51026
I20260925 18:11:29.038435 822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 18:11:29.041364 1545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:29.042125 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:11:29.042366 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/ts-1-root
uuid: "0b51096c8830473ca018f49c58e3a613"
format_stamp: "Formatted at 2026-09-25 18:11:29 on dist-test-slave-57m3"
I20260925 18:11:29.042604 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1790359881141916-822-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:29.071708 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:29.072645 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:29.074009 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:29.076138 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:29.076323 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:29.076571 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:29.076707 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:29.110950 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.130:41097
I20260925 18:11:29.111013 1607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.130:41097 every 8 connection(s)
I20260925 18:11:29.124223 1608 heartbeater.cc:347] Connected to a master server at 127.0.205.190:37917
I20260925 18:11:29.124568 1608 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:29.125180 1608 heartbeater.cc:511] Master 127.0.205.190:37917 requested a full tablet report, sending...
I20260925 18:11:29.126873 1406 ts_manager.cc:194] Registered new tserver with Master: 0b51096c8830473ca018f49c58e3a613 (127.0.205.130:41097)
I20260925 18:11:29.127851 822 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.014165058s
I20260925 18:11:29.128279 1406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51038
I20260925 18:11:29.137583 1533 heartbeater.cc:503] Master 127.0.205.190:37917 was elected leader, sending a full tablet report...
I20260925 18:11:29.148340 1406 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 18:11:29.149564 1406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260925 18:11:29.179051 1573 tablet_service.cc:1467] Processing CreateTablet for tablet d2792baafb83426fad5e7b227cb5aac9 (DEFAULT_TABLE table=test-table [id=5f3a616a4f9c4b4b83dacb273061d9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:29.180063 1573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2792baafb83426fad5e7b227cb5aac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:29.179991 1498 tablet_service.cc:1467] Processing CreateTablet for tablet d2792baafb83426fad5e7b227cb5aac9 (DEFAULT_TABLE table=test-table [id=5f3a616a4f9c4b4b83dacb273061d9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:29.180982 1498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2792baafb83426fad5e7b227cb5aac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:29.191303 1621 tablet_bootstrap.cc:492] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613: Bootstrap starting.
I20260925 18:11:29.192034 1622 tablet_bootstrap.cc:492] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5: Bootstrap starting.
I20260925 18:11:29.195540 1621 tablet_bootstrap.cc:654] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:29.195909 1622 tablet_bootstrap.cc:654] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:29.199687 1622 tablet_bootstrap.cc:492] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5: No bootstrap required, opened a new log
I20260925 18:11:29.199703 1621 tablet_bootstrap.cc:492] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613: No bootstrap required, opened a new log
I20260925 18:11:29.200054 1622 ts_tablet_manager.cc:1397] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 18:11:29.200111 1621 ts_tablet_manager.cc:1397] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 18:11:29.201771 1622 raft_consensus.cc:348] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 } } peers { permanent_uuid: "618e2faf69a645568e3c8d9cf46705a5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 37769 } }
I20260925 18:11:29.201962 1621 raft_consensus.cc:348] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 } } peers { permanent_uuid: "618e2faf69a645568e3c8d9cf46705a5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 37769 } }
I20260925 18:11:29.202227 1622 raft_consensus.cc:729] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618e2faf69a645568e3c8d9cf46705a5, State: Initialized, Role: FOLLOWER
I20260925 18:11:29.202450 1621 raft_consensus.cc:729] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b51096c8830473ca018f49c58e3a613, State: Initialized, Role: FOLLOWER
I20260925 18:11:29.202634 1622 consensus_queue.cc:260] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 } } peers { permanent_uuid: "618e2faf69a645568e3c8d9cf46705a5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 37769 } }
I20260925 18:11:29.202862 1621 consensus_queue.cc:260] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 } } peers { permanent_uuid: "618e2faf69a645568e3c8d9cf46705a5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 37769 } }
I20260925 18:11:29.205711 1622 ts_tablet_manager.cc:1428] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 18:11:29.207160 1608 heartbeater.cc:503] Master 127.0.205.190:37917 was elected leader, sending a full tablet report...
I20260925 18:11:29.207593 1621 ts_tablet_manager.cc:1428] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 18:11:29.231446 822 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260925 18:11:29.231665 822 raft_consensus.cc:412] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 0 FOLLOWER]: Emulating election...
I20260925 18:11:29.231871 822 raft_consensus.cc:3037] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:29.235644 822 raft_consensus.cc:686] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 1 LEADER]: Becoming Leader. State: Replica: 618e2faf69a645568e3c8d9cf46705a5, State: Running, Role: LEADER
W20260925 18:11:29.235922 1533 heartbeater.cc:644] Failed to heartbeat to 127.0.205.190:37917 (0 consecutive failures): IO error: failing all heartbeats for tests
I20260925 18:11:29.236238 822 consensus_queue.cc:237] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 } } peers { permanent_uuid: "618e2faf69a645568e3c8d9cf46705a5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 37769 } }
I20260925 18:11:29.238802 822 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260925 18:11:29.258666 822 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260925 18:11:29.362129 822 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260925 18:11:29.565547 822 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0
I20260925 18:11:29.648867 1583 raft_consensus.cc:3037] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:29.654156 1625 consensus_queue.cc:1035] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:29.870003 822 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260925 18:11:29.870214 822 raft_consensus.cc:412] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 1 LEADER]: Emulating election...
I20260925 18:11:29.870375 822 raft_consensus.cc:3032] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 18:11:29.870517 822 raft_consensus.cc:729] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 618e2faf69a645568e3c8d9cf46705a5, State: Running, Role: LEADER
I20260925 18:11:29.870867 822 consensus_queue.cc:260] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 } } peers { permanent_uuid: "618e2faf69a645568e3c8d9cf46705a5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 37769 } }
I20260925 18:11:29.871435 822 raft_consensus.cc:3037] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:29.875016 822 raft_consensus.cc:686] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 2 LEADER]: Becoming Leader. State: Replica: 618e2faf69a645568e3c8d9cf46705a5, State: Running, Role: LEADER
I20260925 18:11:29.875487 822 consensus_queue.cc:237] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 } } peers { permanent_uuid: "618e2faf69a645568e3c8d9cf46705a5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 37769 } }
I20260925 18:11:29.876720 822 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260925 18:11:29.880065 822 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260925 18:11:29.983632 822 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260925 18:11:30.187132 822 cluster_itest_util.cc:256] Not converged past 2 yet: 2.2 1.1
I20260925 18:11:30.255587 1583 raft_consensus.cc:3037] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:30.259794 1583 raft_consensus.cc:1261] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613 [term 2 FOLLOWER]: Refusing update from remote peer 618e2faf69a645568e3c8d9cf46705a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 18:11:30.260913 1631 consensus_queue.cc:1035] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 18:11:30.491387 822 ts_tablet_manager-itest.cc:367] Electing peer 0...
I20260925 18:11:30.491590 822 raft_consensus.cc:412] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 2 LEADER]: Emulating election...
I20260925 18:11:30.491746 822 raft_consensus.cc:3032] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 18:11:30.491888 822 raft_consensus.cc:729] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 618e2faf69a645568e3c8d9cf46705a5, State: Running, Role: LEADER
I20260925 18:11:30.492231 822 consensus_queue.cc:260] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 } } peers { permanent_uuid: "618e2faf69a645568e3c8d9cf46705a5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 37769 } }
I20260925 18:11:30.492808 822 raft_consensus.cc:3037] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:30.496382 822 raft_consensus.cc:686] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 3 LEADER]: Becoming Leader. State: Replica: 618e2faf69a645568e3c8d9cf46705a5, State: Running, Role: LEADER
I20260925 18:11:30.496920 822 consensus_queue.cc:237] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 } } peers { permanent_uuid: "618e2faf69a645568e3c8d9cf46705a5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 37769 } }
I20260925 18:11:30.498056 822 ts_tablet_manager-itest.cc:370] Waiting for servers to agree...
I20260925 18:11:30.500989 822 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260925 18:11:30.604347 822 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260925 18:11:30.807847 822 cluster_itest_util.cc:256] Not converged past 3 yet: 3.3 2.2
I20260925 18:11:31.035907 1583 raft_consensus.cc:3037] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:31.040164 1583 raft_consensus.cc:1261] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613 [term 3 FOLLOWER]: Refusing update from remote peer 618e2faf69a645568e3c8d9cf46705a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260925 18:11:31.041270 1638 consensus_queue.cc:1035] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b51096c8830473ca018f49c58e3a613" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:11:31.114252 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:11:31.128640 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:31.129166 822 tablet_replica.cc:330] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5: stopping tablet replica
I20260925 18:11:31.129580 822 raft_consensus.cc:2229] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 3 LEADER]: Raft consensus shutting down.
I20260925 18:11:31.130097 822 raft_consensus.cc:2258] T d2792baafb83426fad5e7b227cb5aac9 P 618e2faf69a645568e3c8d9cf46705a5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:31.137293 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:11:31.143730 822 tablet_server.cc:178] TabletServer@127.0.205.130:0 shutting down...
I20260925 18:11:31.156514 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:31.157061 822 tablet_replica.cc:330] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613: stopping tablet replica
I20260925 18:11:31.157497 822 raft_consensus.cc:2229] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260925 18:11:31.157874 822 raft_consensus.cc:2258] T d2792baafb83426fad5e7b227cb5aac9 P 0b51096c8830473ca018f49c58e3a613 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:31.175202 822 tablet_server.cc:194] TabletServer@127.0.205.130:0 shutdown complete.
I20260925 18:11:31.181751 822 master.cc:537] Master@127.0.205.190:37917 shutting down...
I20260925 18:11:31.192278 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:11:31.192715 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:31.193025 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P d36291bd9c624769b8f956b4fa169477: stopping tablet replica
I20260925 18:11:31.208642 822 master.cc:552] Master@127.0.205.190:37917 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2492 ms)
[----------] 3 tests from LeadershipChangeReportingTest (9927 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20260925 18:11:31.230327 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:36451
I20260925 18:11:31.231292 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:31.235991 1647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:31.236811 1648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:31.238611 1650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:31.239006 822 server_base.cc:1034] running on GCE node
I20260925 18:11:31.239635 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:31.239787 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:31.239915 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359891239904 us; error 0 us; skew 500 ppm
I20260925 18:11:31.240394 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:31.242548 822 webserver.cc:460] Webserver started at http://127.0.205.190:39059/ using document root <none> and password file <none>
I20260925 18:11:31.242969 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:31.243157 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:31.243386 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:31.244374 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "0e979efb49e74c5ebd1446be854ed747"
format_stamp: "Formatted at 2026-09-25 18:11:31 on dist-test-slave-57m3"
I20260925 18:11:31.248338 822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 18:11:31.251302 1655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:31.251927 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:11:31.252172 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "0e979efb49e74c5ebd1446be854ed747"
format_stamp: "Formatted at 2026-09-25 18:11:31 on dist-test-slave-57m3"
I20260925 18:11:31.252421 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:31.289865 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:31.290818 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:31.338315 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:36451
I20260925 18:11:31.338403 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:36451 every 8 connection(s)
I20260925 18:11:31.342731 1708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:31.352901 1708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747: Bootstrap starting.
I20260925 18:11:31.357311 1708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:31.361227 1708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747: No bootstrap required, opened a new log
I20260925 18:11:31.363171 1708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e979efb49e74c5ebd1446be854ed747" member_type: VOTER }
I20260925 18:11:31.363665 1708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:31.363852 1708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e979efb49e74c5ebd1446be854ed747, State: Initialized, Role: FOLLOWER
I20260925 18:11:31.364310 1708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e979efb49e74c5ebd1446be854ed747" member_type: VOTER }
I20260925 18:11:31.364693 1708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:11:31.364923 1708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:11:31.365149 1708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:31.368883 1708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e979efb49e74c5ebd1446be854ed747" member_type: VOTER }
I20260925 18:11:31.369329 1708 leader_election.cc:302] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e979efb49e74c5ebd1446be854ed747; no voters:
I20260925 18:11:31.370321 1708 leader_election.cc:288] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:11:31.370671 1711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:11:31.371829 1711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 1 LEADER]: Becoming Leader. State: Replica: 0e979efb49e74c5ebd1446be854ed747, State: Running, Role: LEADER
I20260925 18:11:31.372361 1711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e979efb49e74c5ebd1446be854ed747" member_type: VOTER }
I20260925 18:11:31.373060 1708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:11:31.375303 1713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e979efb49e74c5ebd1446be854ed747. Latest consensus state: current_term: 1 leader_uuid: "0e979efb49e74c5ebd1446be854ed747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e979efb49e74c5ebd1446be854ed747" member_type: VOTER } }
I20260925 18:11:31.375277 1712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e979efb49e74c5ebd1446be854ed747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e979efb49e74c5ebd1446be854ed747" member_type: VOTER } }
I20260925 18:11:31.375973 1713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:31.376161 1712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:31.378551 1717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:11:31.384508 1717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:11:31.386520 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:11:31.393949 1717 catalog_manager.cc:1347] Generated new cluster ID: 360e5637a13842f78eb47d35d56275fe
I20260925 18:11:31.394289 1717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:11:31.412577 1717 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:11:31.414098 1717 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:11:31.438175 1717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747: Generated new TSK 0
I20260925 18:11:31.438759 1717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:11:31.452935 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:31.459671 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:31.461023 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:31.462132 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:31.463064 822 server_base.cc:1034] running on GCE node
I20260925 18:11:31.463829 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:31.464004 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:31.464148 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359891464135 us; error 0 us; skew 500 ppm
I20260925 18:11:31.464649 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:31.466965 822 webserver.cc:460] Webserver started at http://127.0.205.129:35729/ using document root <none> and password file <none>
I20260925 18:11:31.467415 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:31.467594 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:31.467875 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:31.469121 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "b1fd2e17ad5146a49aa8a3831ddce22b"
format_stamp: "Formatted at 2026-09-25 18:11:31 on dist-test-slave-57m3"
I20260925 18:11:31.473483 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 18:11:31.476603 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:31.477342 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:11:31.477581 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "b1fd2e17ad5146a49aa8a3831ddce22b"
format_stamp: "Formatted at 2026-09-25 18:11:31 on dist-test-slave-57m3"
I20260925 18:11:31.477818 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:31.493034 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:31.494186 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:31.495494 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:31.497831 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:31.498025 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:31.498255 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:31.498394 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:31.534866 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:45889
I20260925 18:11:31.534951 1799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:45889 every 8 connection(s)
I20260925 18:11:31.539361 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:31.545804 1804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:31.547119 1805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:31.550438 1807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:31.550863 822 server_base.cc:1034] running on GCE node
I20260925 18:11:31.551688 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:31.551927 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:31.552106 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359891552093 us; error 0 us; skew 500 ppm
I20260925 18:11:31.552814 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:31.553807 1800 heartbeater.cc:347] Connected to a master server at 127.0.205.190:36451
I20260925 18:11:31.554227 1800 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:31.554848 1800 heartbeater.cc:511] Master 127.0.205.190:36451 requested a full tablet report, sending...
I20260925 18:11:31.555516 822 webserver.cc:460] Webserver started at http://127.0.205.130:34311/ using document root <none> and password file <none>
I20260925 18:11:31.556066 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:31.556241 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:31.556490 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:31.556879 1673 ts_manager.cc:194] Registered new tserver with Master: b1fd2e17ad5146a49aa8a3831ddce22b (127.0.205.129:45889)
I20260925 18:11:31.557699 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-1-root/instance:
uuid: "27063a44bf484cceac5fa5fba1f990ba"
format_stamp: "Formatted at 2026-09-25 18:11:31 on dist-test-slave-57m3"
I20260925 18:11:31.558996 1673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45878
I20260925 18:11:31.562405 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 18:11:31.565546 1812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:31.566289 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260925 18:11:31.566530 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-1-root
uuid: "27063a44bf484cceac5fa5fba1f990ba"
format_stamp: "Formatted at 2026-09-25 18:11:31 on dist-test-slave-57m3"
I20260925 18:11:31.566771 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:31.577768 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:31.578891 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:31.580229 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:31.582384 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:31.582561 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:31.582756 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:31.582899 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:31.618777 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.130:46121
I20260925 18:11:31.618897 1874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.130:46121 every 8 connection(s)
I20260925 18:11:31.623025 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:31.630016 1878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:31.630895 1879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:31.632172 1881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:31.633028 822 server_base.cc:1034] running on GCE node
I20260925 18:11:31.633781 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:31.633970 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:31.634115 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359891634101 us; error 0 us; skew 500 ppm
I20260925 18:11:31.634593 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:31.635879 1875 heartbeater.cc:347] Connected to a master server at 127.0.205.190:36451
I20260925 18:11:31.636305 1875 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:31.637166 1875 heartbeater.cc:511] Master 127.0.205.190:36451 requested a full tablet report, sending...
I20260925 18:11:31.637441 822 webserver.cc:460] Webserver started at http://127.0.205.131:36383/ using document root <none> and password file <none>
I20260925 18:11:31.638024 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:31.638258 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:31.638558 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:31.639206 1673 ts_manager.cc:194] Registered new tserver with Master: 27063a44bf484cceac5fa5fba1f990ba (127.0.205.130:46121)
I20260925 18:11:31.639791 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-2-root/instance:
uuid: "cb44e5ff4b274b24b2efaea38b28c307"
format_stamp: "Formatted at 2026-09-25 18:11:31 on dist-test-slave-57m3"
I20260925 18:11:31.640676 1673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45888
I20260925 18:11:31.644240 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 18:11:31.647291 1886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:31.647915 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:11:31.648171 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-2-root
uuid: "cb44e5ff4b274b24b2efaea38b28c307"
format_stamp: "Formatted at 2026-09-25 18:11:31 on dist-test-slave-57m3"
I20260925 18:11:31.648435 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1790359881141916-822-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:31.659201 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:31.660310 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:31.661599 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:31.663681 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:31.663854 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:31.664058 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:31.664214 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:31.726858 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.131:35111
I20260925 18:11:31.726907 1948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.131:35111 every 8 connection(s)
I20260925 18:11:31.741967 1949 heartbeater.cc:347] Connected to a master server at 127.0.205.190:36451
I20260925 18:11:31.742321 1949 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:31.743063 1949 heartbeater.cc:511] Master 127.0.205.190:36451 requested a full tablet report, sending...
I20260925 18:11:31.744778 1673 ts_manager.cc:194] Registered new tserver with Master: cb44e5ff4b274b24b2efaea38b28c307 (127.0.205.131:35111)
I20260925 18:11:31.745272 822 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014527781s
I20260925 18:11:31.746253 1673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35474
I20260925 18:11:31.770304 1673 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260925 18:11:31.771763 1673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 18:11:31.815935 1914 tablet_service.cc:1467] Processing CreateTablet for tablet f80f31bb72094a088399eda41fa7055c (DEFAULT_TABLE table=test-table [id=9166d3cec8864b40bf759ac3b432d1da]), partition=
I20260925 18:11:31.817090 1914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80f31bb72094a088399eda41fa7055c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:31.819041 1840 tablet_service.cc:1467] Processing CreateTablet for tablet f80f31bb72094a088399eda41fa7055c (DEFAULT_TABLE table=test-table [id=9166d3cec8864b40bf759ac3b432d1da]), partition=
I20260925 18:11:31.820040 1840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80f31bb72094a088399eda41fa7055c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:31.822168 1765 tablet_service.cc:1467] Processing CreateTablet for tablet f80f31bb72094a088399eda41fa7055c (DEFAULT_TABLE table=test-table [id=9166d3cec8864b40bf759ac3b432d1da]), partition=
I20260925 18:11:31.823114 1765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80f31bb72094a088399eda41fa7055c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:31.834618 1964 tablet_bootstrap.cc:492] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307: Bootstrap starting.
I20260925 18:11:31.839435 1966 tablet_bootstrap.cc:492] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba: Bootstrap starting.
I20260925 18:11:31.840718 1965 tablet_bootstrap.cc:492] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b: Bootstrap starting.
I20260925 18:11:31.841892 1964 tablet_bootstrap.cc:654] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:31.845073 1966 tablet_bootstrap.cc:654] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:31.846637 1965 tablet_bootstrap.cc:654] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:31.848047 1964 tablet_bootstrap.cc:492] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307: No bootstrap required, opened a new log
I20260925 18:11:31.848490 1964 ts_tablet_manager.cc:1397] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260925 18:11:31.850907 1966 tablet_bootstrap.cc:492] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba: No bootstrap required, opened a new log
I20260925 18:11:31.851317 1966 ts_tablet_manager.cc:1397] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260925 18:11:31.851348 1964 raft_consensus.cc:348] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:31.852058 1964 raft_consensus.cc:374] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:31.852349 1964 raft_consensus.cc:729] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb44e5ff4b274b24b2efaea38b28c307, State: Initialized, Role: FOLLOWER
I20260925 18:11:31.853363 1965 tablet_bootstrap.cc:492] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b: No bootstrap required, opened a new log
I20260925 18:11:31.853096 1964 consensus_queue.cc:260] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:31.853749 1965 ts_tablet_manager.cc:1397] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20260925 18:11:31.854713 1966 raft_consensus.cc:348] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:31.855499 1966 raft_consensus.cc:374] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:31.855880 1966 raft_consensus.cc:729] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27063a44bf484cceac5fa5fba1f990ba, State: Initialized, Role: FOLLOWER
I20260925 18:11:31.856766 1949 heartbeater.cc:503] Master 127.0.205.190:36451 was elected leader, sending a full tablet report...
I20260925 18:11:31.857028 1964 ts_tablet_manager.cc:1428] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307: Time spent starting tablet: real 0.008s user 0.007s sys 0.003s
I20260925 18:11:31.857120 1966 consensus_queue.cc:260] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:31.858585 1965 raft_consensus.cc:348] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:31.859344 1965 raft_consensus.cc:374] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:31.859637 1965 raft_consensus.cc:729] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1fd2e17ad5146a49aa8a3831ddce22b, State: Initialized, Role: FOLLOWER
I20260925 18:11:31.860252 1875 heartbeater.cc:503] Master 127.0.205.190:36451 was elected leader, sending a full tablet report...
I20260925 18:11:31.860323 1965 consensus_queue.cc:260] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:31.861061 1966 ts_tablet_manager.cc:1428] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba: Time spent starting tablet: real 0.009s user 0.002s sys 0.008s
I20260925 18:11:31.862455 1800 heartbeater.cc:503] Master 127.0.205.190:36451 was elected leader, sending a full tablet report...
I20260925 18:11:31.863072 1965 ts_tablet_manager.cc:1428] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260925 18:11:32.003713 1972 raft_consensus.cc:482] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:32.004117 1972 raft_consensus.cc:504] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:32.006108 1972 leader_election.cc:288] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb44e5ff4b274b24b2efaea38b28c307 (127.0.205.131:35111), 27063a44bf484cceac5fa5fba1f990ba (127.0.205.130:46121)
I20260925 18:11:32.017644 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f31bb72094a088399eda41fa7055c" candidate_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb44e5ff4b274b24b2efaea38b28c307" is_pre_election: true
I20260925 18:11:32.018627 1924 raft_consensus.cc:2454] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1fd2e17ad5146a49aa8a3831ddce22b in term 0.
I20260925 18:11:32.018841 1850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f31bb72094a088399eda41fa7055c" candidate_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27063a44bf484cceac5fa5fba1f990ba" is_pre_election: true
I20260925 18:11:32.019408 1850 raft_consensus.cc:2454] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1fd2e17ad5146a49aa8a3831ddce22b in term 0.
I20260925 18:11:32.019598 1741 leader_election.cc:302] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1fd2e17ad5146a49aa8a3831ddce22b, cb44e5ff4b274b24b2efaea38b28c307; no voters:
I20260925 18:11:32.020325 1972 raft_consensus.cc:2781] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 18:11:32.020586 1972 raft_consensus.cc:482] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:11:32.020828 1972 raft_consensus.cc:3037] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:32.025007 1972 raft_consensus.cc:504] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:32.026325 1972 leader_election.cc:288] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [CANDIDATE]: Term 1 election: Requested vote from peers cb44e5ff4b274b24b2efaea38b28c307 (127.0.205.131:35111), 27063a44bf484cceac5fa5fba1f990ba (127.0.205.130:46121)
I20260925 18:11:32.027132 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f31bb72094a088399eda41fa7055c" candidate_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb44e5ff4b274b24b2efaea38b28c307"
I20260925 18:11:32.027318 1850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f31bb72094a088399eda41fa7055c" candidate_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27063a44bf484cceac5fa5fba1f990ba"
I20260925 18:11:32.027715 1924 raft_consensus.cc:3037] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:32.027802 1850 raft_consensus.cc:3037] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:32.034176 1924 raft_consensus.cc:2454] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1fd2e17ad5146a49aa8a3831ddce22b in term 1.
I20260925 18:11:32.034174 1850 raft_consensus.cc:2454] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1fd2e17ad5146a49aa8a3831ddce22b in term 1.
I20260925 18:11:32.035439 1740 leader_election.cc:302] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27063a44bf484cceac5fa5fba1f990ba, b1fd2e17ad5146a49aa8a3831ddce22b; no voters:
I20260925 18:11:32.036067 1972 raft_consensus.cc:2781] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:11:32.036821 1972 raft_consensus.cc:686] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 1 LEADER]: Becoming Leader. State: Replica: b1fd2e17ad5146a49aa8a3831ddce22b, State: Running, Role: LEADER
I20260925 18:11:32.037611 1972 consensus_queue.cc:237] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:32.043366 1673 catalog_manager.cc:5495] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b reported cstate change: term changed from 0 to 1, leader changed from <none> to b1fd2e17ad5146a49aa8a3831ddce22b (127.0.205.129). New cstate: current_term: 1 leader_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } health_report { overall_health: HEALTHY } } }
I20260925 18:11:32.567482 1977 consensus_queue.cc:1035] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:32.578141 1980 consensus_queue.cc:1035] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [LEADER]: Connected to new peer: Peer: permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:33.152755 822 tablet_replica.cc:330] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b: stopping tablet replica
I20260925 18:11:33.153333 822 raft_consensus.cc:2229] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:11:33.154002 822 raft_consensus.cc:2258] T f80f31bb72094a088399eda41fa7055c P b1fd2e17ad5146a49aa8a3831ddce22b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:34.478607 1987 raft_consensus.cc:482] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b1fd2e17ad5146a49aa8a3831ddce22b)
I20260925 18:11:34.479055 1987 raft_consensus.cc:504] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:34.480899 1987 leader_election.cc:288] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27063a44bf484cceac5fa5fba1f990ba (127.0.205.130:46121), b1fd2e17ad5146a49aa8a3831ddce22b (127.0.205.129:45889)
I20260925 18:11:34.501896 1992 raft_consensus.cc:482] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b1fd2e17ad5146a49aa8a3831ddce22b)
I20260925 18:11:34.502657 1992 raft_consensus.cc:504] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:34.504707 1850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f31bb72094a088399eda41fa7055c" candidate_uuid: "cb44e5ff4b274b24b2efaea38b28c307" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27063a44bf484cceac5fa5fba1f990ba" is_pre_election: true
I20260925 18:11:34.505256 1850 raft_consensus.cc:2454] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb44e5ff4b274b24b2efaea38b28c307 in term 1.
I20260925 18:11:34.505434 1992 leader_election.cc:288] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cb44e5ff4b274b24b2efaea38b28c307 (127.0.205.131:35111), b1fd2e17ad5146a49aa8a3831ddce22b (127.0.205.129:45889)
I20260925 18:11:34.506368 1889 leader_election.cc:302] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27063a44bf484cceac5fa5fba1f990ba, cb44e5ff4b274b24b2efaea38b28c307; no voters:
I20260925 18:11:34.507294 1987 raft_consensus.cc:2781] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 18:11:34.507629 1987 raft_consensus.cc:482] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b1fd2e17ad5146a49aa8a3831ddce22b)
I20260925 18:11:34.507573 1775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f31bb72094a088399eda41fa7055c" candidate_uuid: "cb44e5ff4b274b24b2efaea38b28c307" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" is_pre_election: true
I20260925 18:11:34.508060 1987 raft_consensus.cc:3037] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 1 FOLLOWER]: Advancing to term 2
W20260925 18:11:34.509196 1890 leader_election.cc:341] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1fd2e17ad5146a49aa8a3831ddce22b (127.0.205.129:45889): Illegal state: cannot vote while shut down
I20260925 18:11:34.513042 1987 raft_consensus.cc:504] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:34.515512 1987 leader_election.cc:288] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [CANDIDATE]: Term 2 election: Requested vote from peers 27063a44bf484cceac5fa5fba1f990ba (127.0.205.130:46121), b1fd2e17ad5146a49aa8a3831ddce22b (127.0.205.129:45889)
I20260925 18:11:34.515444 1850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f31bb72094a088399eda41fa7055c" candidate_uuid: "cb44e5ff4b274b24b2efaea38b28c307" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27063a44bf484cceac5fa5fba1f990ba"
I20260925 18:11:34.516072 1850 raft_consensus.cc:3037] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:34.516476 1775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f31bb72094a088399eda41fa7055c" candidate_uuid: "cb44e5ff4b274b24b2efaea38b28c307" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b"
W20260925 18:11:34.517549 1890 leader_election.cc:341] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b1fd2e17ad5146a49aa8a3831ddce22b (127.0.205.129:45889): Illegal state: cannot vote while shut down
I20260925 18:11:34.522513 1850 raft_consensus.cc:2454] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb44e5ff4b274b24b2efaea38b28c307 in term 2.
I20260925 18:11:34.524659 1889 leader_election.cc:302] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27063a44bf484cceac5fa5fba1f990ba, cb44e5ff4b274b24b2efaea38b28c307; no voters: b1fd2e17ad5146a49aa8a3831ddce22b
I20260925 18:11:34.525700 1987 raft_consensus.cc:2781] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 18:11:34.526343 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f31bb72094a088399eda41fa7055c" candidate_uuid: "27063a44bf484cceac5fa5fba1f990ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cb44e5ff4b274b24b2efaea38b28c307" is_pre_election: true
I20260925 18:11:34.526973 1987 raft_consensus.cc:686] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 2 LEADER]: Becoming Leader. State: Replica: cb44e5ff4b274b24b2efaea38b28c307, State: Running, Role: LEADER
I20260925 18:11:34.527711 1987 consensus_queue.cc:237] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb44e5ff4b274b24b2efaea38b28c307" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 35111 } } peers { permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 } } peers { permanent_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45889 } }
I20260925 18:11:34.532545 1775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80f31bb72094a088399eda41fa7055c" candidate_uuid: "27063a44bf484cceac5fa5fba1f990ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b1fd2e17ad5146a49aa8a3831ddce22b" is_pre_election: true
W20260925 18:11:34.533735 1816 leader_election.cc:341] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1fd2e17ad5146a49aa8a3831ddce22b (127.0.205.129:45889): Illegal state: cannot vote while shut down
I20260925 18:11:34.534648 1816 leader_election.cc:302] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27063a44bf484cceac5fa5fba1f990ba; no voters: b1fd2e17ad5146a49aa8a3831ddce22b, cb44e5ff4b274b24b2efaea38b28c307
I20260925 18:11:34.535265 1992 raft_consensus.cc:2726] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 18:11:34.975239 1850 raft_consensus.cc:1261] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 2 FOLLOWER]: Refusing update from remote peer cb44e5ff4b274b24b2efaea38b28c307: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 18:11:34.976361 1987 consensus_queue.cc:1035] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27063a44bf484cceac5fa5fba1f990ba" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 18:11:34.985306 1890 consensus_queue.cc:579] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [LEADER]: The tablet replica hosted on peer b1fd2e17ad5146a49aa8a3831ddce22b has failed
W20260925 18:11:34.985828 1890 consensus_peers.cc:487] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 -> Peer b1fd2e17ad5146a49aa8a3831ddce22b (127.0.205.129:45889): Couldn't send request to peer b1fd2e17ad5146a49aa8a3831ddce22b. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260925 18:11:35.187496 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:11:35.201583 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:35.218376 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:11:35.225651 822 tablet_server.cc:178] TabletServer@127.0.205.130:0 shutting down...
I20260925 18:11:35.237975 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:35.238616 822 tablet_replica.cc:330] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba: stopping tablet replica
I20260925 18:11:35.239113 822 raft_consensus.cc:2229] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 18:11:35.239492 822 raft_consensus.cc:2258] T f80f31bb72094a088399eda41fa7055c P 27063a44bf484cceac5fa5fba1f990ba [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:35.257153 822 tablet_server.cc:194] TabletServer@127.0.205.130:0 shutdown complete.
I20260925 18:11:35.264104 822 tablet_server.cc:178] TabletServer@127.0.205.131:0 shutting down...
I20260925 18:11:35.276901 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:35.277436 822 tablet_replica.cc:330] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307: stopping tablet replica
I20260925 18:11:35.277882 822 raft_consensus.cc:2229] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 2 LEADER]: Raft consensus shutting down.
I20260925 18:11:35.278555 822 raft_consensus.cc:2258] T f80f31bb72094a088399eda41fa7055c P cb44e5ff4b274b24b2efaea38b28c307 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:35.296444 822 tablet_server.cc:194] TabletServer@127.0.205.131:0 shutdown complete.
I20260925 18:11:35.304759 822 master.cc:537] Master@127.0.205.190:36451 shutting down...
I20260925 18:11:35.318493 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:11:35.318923 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:35.319157 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0e979efb49e74c5ebd1446be854ed747: stopping tablet replica
I20260925 18:11:35.335129 822 master.cc:552] Master@127.0.205.190:36451 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (4129 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20260925 18:11:35.360198 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:33459
I20260925 18:11:35.361192 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:35.365600 2006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:35.366575 2007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:35.368518 2009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:35.368681 822 server_base.cc:1034] running on GCE node
I20260925 18:11:35.369451 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:35.369619 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:35.369752 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359895369738 us; error 0 us; skew 500 ppm
I20260925 18:11:35.370208 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:35.378228 822 webserver.cc:460] Webserver started at http://127.0.205.190:33177/ using document root <none> and password file <none>
I20260925 18:11:35.378703 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:35.378870 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:35.379106 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:35.380103 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "aa33cf47cfc64e718aba818607611c81"
format_stamp: "Formatted at 2026-09-25 18:11:35 on dist-test-slave-57m3"
I20260925 18:11:35.384248 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 18:11:35.387310 2014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:35.388072 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:11:35.388307 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "aa33cf47cfc64e718aba818607611c81"
format_stamp: "Formatted at 2026-09-25 18:11:35 on dist-test-slave-57m3"
I20260925 18:11:35.388540 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:35.401254 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:35.402272 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:35.437027 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:33459
I20260925 18:11:35.437075 2065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:33459 every 8 connection(s)
I20260925 18:11:35.440439 2066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:35.449676 2066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81: Bootstrap starting.
I20260925 18:11:35.453352 2066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:35.456923 2066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81: No bootstrap required, opened a new log
I20260925 18:11:35.458736 2066 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa33cf47cfc64e718aba818607611c81" member_type: VOTER }
I20260925 18:11:35.459134 2066 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:35.459355 2066 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa33cf47cfc64e718aba818607611c81, State: Initialized, Role: FOLLOWER
I20260925 18:11:35.459897 2066 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa33cf47cfc64e718aba818607611c81" member_type: VOTER }
I20260925 18:11:35.460314 2066 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:11:35.460527 2066 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:11:35.460762 2066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:35.464653 2066 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa33cf47cfc64e718aba818607611c81" member_type: VOTER }
I20260925 18:11:35.465132 2066 leader_election.cc:302] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa33cf47cfc64e718aba818607611c81; no voters:
I20260925 18:11:35.466140 2066 leader_election.cc:288] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:11:35.466398 2069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:11:35.467599 2069 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 1 LEADER]: Becoming Leader. State: Replica: aa33cf47cfc64e718aba818607611c81, State: Running, Role: LEADER
I20260925 18:11:35.468110 2069 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa33cf47cfc64e718aba818607611c81" member_type: VOTER }
I20260925 18:11:35.468736 2066 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:11:35.470710 2071 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa33cf47cfc64e718aba818607611c81. Latest consensus state: current_term: 1 leader_uuid: "aa33cf47cfc64e718aba818607611c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa33cf47cfc64e718aba818607611c81" member_type: VOTER } }
I20260925 18:11:35.470774 2070 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa33cf47cfc64e718aba818607611c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa33cf47cfc64e718aba818607611c81" member_type: VOTER } }
I20260925 18:11:35.471450 2070 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:35.472820 2071 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:35.474107 2075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:11:35.478374 2075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:11:35.482441 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:11:35.485824 2075 catalog_manager.cc:1347] Generated new cluster ID: 7bd93aa244864f4bbcf3cc4a3dd7ae48
I20260925 18:11:35.486092 2075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:11:35.516567 2075 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:11:35.518162 2075 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:11:35.531520 2075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81: Generated new TSK 0
I20260925 18:11:35.532150 2075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:11:35.548965 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:35.554019 2087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:35.555002 2088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:35.556857 822 server_base.cc:1034] running on GCE node
W20260925 18:11:35.557232 2090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:35.558059 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:35.558224 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:35.558346 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359895558334 us; error 0 us; skew 500 ppm
I20260925 18:11:35.558822 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:35.560930 822 webserver.cc:460] Webserver started at http://127.0.205.129:34237/ using document root <none> and password file <none>
I20260925 18:11:35.561360 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:35.561528 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:35.561765 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:35.562812 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "fa2e41b6ec764b5bbdd5129d94bece2e"
format_stamp: "Formatted at 2026-09-25 18:11:35 on dist-test-slave-57m3"
I20260925 18:11:35.566898 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 18:11:35.569900 2095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:35.570700 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:11:35.570948 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "fa2e41b6ec764b5bbdd5129d94bece2e"
format_stamp: "Formatted at 2026-09-25 18:11:35 on dist-test-slave-57m3"
I20260925 18:11:35.571206 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:35.579131 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:35.580119 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:35.581424 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:35.583467 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:35.583639 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:35.583842 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:35.583985 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:35.617707 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:42899
I20260925 18:11:35.617784 2157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:42899 every 8 connection(s)
I20260925 18:11:35.631860 2158 heartbeater.cc:347] Connected to a master server at 127.0.205.190:33459
I20260925 18:11:35.632170 2158 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:35.632766 2158 heartbeater.cc:511] Master 127.0.205.190:33459 requested a full tablet report, sending...
I20260925 18:11:35.634445 2031 ts_manager.cc:194] Registered new tserver with Master: fa2e41b6ec764b5bbdd5129d94bece2e (127.0.205.129:42899)
I20260925 18:11:35.634523 822 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.014032126s
I20260925 18:11:35.636248 2031 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58702
I20260925 18:11:35.653633 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:11:35.665423 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:35.679844 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:11:35.686486 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 18:11:35.688998 822 tablet_server.cc:93] deduplicated master addresses: 127.0.205.190:33459
W20260925 18:11:35.691867 2168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:35.693279 2169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:35.693307 2171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:35.694018 822 server_base.cc:1034] running on GCE node
I20260925 18:11:35.694694 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:35.694875 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:35.695016 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359895695001 us; error 0 us; skew 500 ppm
I20260925 18:11:35.695480 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:35.697590 822 webserver.cc:460] Webserver started at http://127.0.205.129:34237/ using document root <none> and password file <none>
I20260925 18:11:35.698036 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:35.698187 822 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260925 18:11:35.701324 822 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 18:11:35.703554 2176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:35.704190 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 18:11:35.704427 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "fa2e41b6ec764b5bbdd5129d94bece2e"
format_stamp: "Formatted at 2026-09-25 18:11:35 on dist-test-slave-57m3"
I20260925 18:11:35.704710 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:35.725819 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:35.726747 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:35.728093 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:35.730165 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:35.730340 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:35.730551 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:35.730683 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:35.765007 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:42899
I20260925 18:11:35.765082 2238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:42899 every 8 connection(s)
I20260925 18:11:35.771353 822 tablet_server.cc:178] TabletServer@127.0.205.129:42899 shutting down...
I20260925 18:11:35.784314 2239 heartbeater.cc:347] Connected to a master server at 127.0.205.190:33459
I20260925 18:11:35.784801 2239 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:35.785486 2239 heartbeater.cc:511] Master 127.0.205.190:33459 requested a full tablet report, sending...
I20260925 18:11:35.787348 2031 ts_manager.cc:194] Re-registered known tserver with Master: fa2e41b6ec764b5bbdd5129d94bece2e (127.0.205.129:42899)
I20260925 18:11:35.788717 2031 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58724
I20260925 18:11:35.791117 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:11:35.805738 822 tablet_server.cc:194] TabletServer@127.0.205.129:42899 shutdown complete.
I20260925 18:11:35.811779 822 master.cc:537] Master@127.0.205.190:33459 shutting down...
I20260925 18:11:35.822899 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:11:35.823357 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:35.823733 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P aa33cf47cfc64e718aba818607611c81: stopping tablet replica
I20260925 18:11:35.840719 822 master.cc:552] Master@127.0.205.190:33459 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (499 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20260925 18:11:35.859724 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:35227,127.0.205.189:37349,127.0.205.188:44299
I20260925 18:11:35.860900 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:35.865218 2246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:35.866145 2247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:35.867414 2249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:35.867648 822 server_base.cc:1034] running on GCE node
I20260925 18:11:35.868306 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:35.868468 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:35.868613 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359895868598 us; error 0 us; skew 500 ppm
I20260925 18:11:35.869153 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:35.871232 822 webserver.cc:460] Webserver started at http://127.0.205.190:44941/ using document root <none> and password file <none>
I20260925 18:11:35.871663 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:35.871834 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:35.872061 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:35.873106 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "c2641b1ebada4ee38658e3bf1fa31415"
format_stamp: "Formatted at 2026-09-25 18:11:35 on dist-test-slave-57m3"
I20260925 18:11:35.877202 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260925 18:11:35.880386 2254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:35.881095 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:11:35.881368 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "c2641b1ebada4ee38658e3bf1fa31415"
format_stamp: "Formatted at 2026-09-25 18:11:35 on dist-test-slave-57m3"
I20260925 18:11:35.881672 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:35.901974 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:35.902944 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:35.934360 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:35227
I20260925 18:11:35.934439 2305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:35227 every 8 connection(s)
I20260925 18:11:35.937816 2306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:35.937914 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:35.942735 2308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:35.944409 2309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:35.944489 2306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } has no permanent_uuid. Determining permanent_uuid...
W20260925 18:11:35.947163 2311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:35.947799 822 server_base.cc:1034] running on GCE node
I20260925 18:11:35.949014 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:35.949263 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:35.949446 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359895949431 us; error 0 us; skew 500 ppm
I20260925 18:11:35.950066 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:35.952565 822 webserver.cc:460] Webserver started at http://127.0.205.189:45941/ using document root <none> and password file <none>
I20260925 18:11:35.953037 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:35.953193 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:35.953464 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:35.954591 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-1-root/instance:
uuid: "9d2456837bc6407e9e32d5503b0f4870"
format_stamp: "Formatted at 2026-09-25 18:11:35 on dist-test-slave-57m3"
I20260925 18:11:35.959627 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 18:11:35.959674 2306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } has no permanent_uuid. Determining permanent_uuid...
W20260925 18:11:35.961737 2255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.205.189:37349: connect: Connection refused (error 111)
I20260925 18:11:35.963635 2319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:35.964355 822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 18:11:35.964604 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-1-root
uuid: "9d2456837bc6407e9e32d5503b0f4870"
format_stamp: "Formatted at 2026-09-25 18:11:35 on dist-test-slave-57m3"
W20260925 18:11:35.964695 2306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.205.189:37349: Network error: Client connection negotiation failed: client connection to 127.0.205.189:37349: connect: Connection refused (error 111)
I20260925 18:11:35.964989 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:35.982376 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:35.983372 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:36.003261 2306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } attempt: 1
W20260925 18:11:36.007720 2306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.205.189:37349: Network error: Client connection negotiation failed: client connection to 127.0.205.189:37349: connect: Connection refused (error 111)
I20260925 18:11:36.016387 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.189:37349
I20260925 18:11:36.016471 2370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.189:37349 every 8 connection(s)
I20260925 18:11:36.019853 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 18:11:36.020012 2371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20260925 18:11:36.024856 2373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:36.026530 2374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:36.027371 2371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } has no permanent_uuid. Determining permanent_uuid...
I20260925 18:11:36.029213 822 server_base.cc:1034] running on GCE node
W20260925 18:11:36.029887 2376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:36.030777 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:36.031028 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:36.031219 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359896031203 us; error 0 us; skew 500 ppm
I20260925 18:11:36.031777 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:36.034453 822 webserver.cc:460] Webserver started at http://127.0.205.188:46517/ using document root <none> and password file <none>
I20260925 18:11:36.035027 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:36.035254 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:36.035537 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:36.036815 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-2-root/instance:
uuid: "fc858f182cf448a3a94ba08cb41d69c0"
format_stamp: "Formatted at 2026-09-25 18:11:36 on dist-test-slave-57m3"
I20260925 18:11:36.042423 822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 18:11:36.043954 2371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } has no permanent_uuid. Determining permanent_uuid...
I20260925 18:11:36.047283 2382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:36.048111 822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260925 18:11:36.048161 2306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } attempt: 2
I20260925 18:11:36.048554 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-2-root
uuid: "fc858f182cf448a3a94ba08cb41d69c0"
format_stamp: "Formatted at 2026-09-25 18:11:36 on dist-test-slave-57m3"
I20260925 18:11:36.048964 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:36.060169 2371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
I20260925 18:11:36.064116 2306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
W20260925 18:11:36.065589 2371 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.205.188:44299: Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111)
W20260925 18:11:36.067536 2306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.205.188:44299: Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111)
I20260925 18:11:36.077714 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:36.078616 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:36.101045 2306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } attempt: 1
I20260925 18:11:36.101059 2371 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } attempt: 1
W20260925 18:11:36.105404 2306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.205.188:44299: Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111)
W20260925 18:11:36.105911 2371 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.205.188:44299: Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111)
I20260925 18:11:36.112434 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.188:44299
I20260925 18:11:36.112514 2436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.188:44299 every 8 connection(s)
I20260925 18:11:36.115123 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:11:36.116089 2437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:36.121652 2437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } has no permanent_uuid. Determining permanent_uuid...
I20260925 18:11:36.132579 2437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } has no permanent_uuid. Determining permanent_uuid...
I20260925 18:11:36.142304 2437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
I20260925 18:11:36.144306 2371 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } attempt: 2
I20260925 18:11:36.160588 2437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: Bootstrap starting.
I20260925 18:11:36.160698 2371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870: Bootstrap starting.
I20260925 18:11:36.164355 2371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:36.164355 2437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:36.168525 2437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: No bootstrap required, opened a new log
I20260925 18:11:36.168529 2371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870: No bootstrap required, opened a new log
I20260925 18:11:36.170375 2437 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:36.170531 2371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:36.170877 2437 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:36.171056 2371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:36.171142 2437 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc858f182cf448a3a94ba08cb41d69c0, State: Initialized, Role: FOLLOWER
I20260925 18:11:36.171334 2371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2456837bc6407e9e32d5503b0f4870, State: Initialized, Role: FOLLOWER
I20260925 18:11:36.171788 2437 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:36.171948 2371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:36.173555 2444 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:36.174399 2444 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:36.175122 2437 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:11:36.173902 2445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:36.175458 2371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:11:36.175588 2445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:36.175945 2446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:36.176375 2446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:36.178403 2446 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:11:36.179874 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0" is_pre_election: true
I20260925 18:11:36.180540 2412 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2456837bc6407e9e32d5503b0f4870 in term 0.
I20260925 18:11:36.181671 2306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } attempt: 2
I20260925 18:11:36.182400 2320 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2456837bc6407e9e32d5503b0f4870, fc858f182cf448a3a94ba08cb41d69c0; no voters:
I20260925 18:11:36.183286 2446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 18:11:36.183606 2446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:11:36.183908 2446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:36.191978 2446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:36.196801 2446 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 1 election: Requested vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:11:36.198261 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0"
I20260925 18:11:36.198922 2412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:36.209811 2306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415: Bootstrap starting.
I20260925 18:11:36.211025 2412 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2456837bc6407e9e32d5503b0f4870 in term 1.
I20260925 18:11:36.213497 2320 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2456837bc6407e9e32d5503b0f4870, fc858f182cf448a3a94ba08cb41d69c0; no voters:
I20260925 18:11:36.214745 2446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:11:36.215214 2446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 1 LEADER]: Becoming Leader. State: Replica: 9d2456837bc6407e9e32d5503b0f4870, State: Running, Role: LEADER
I20260925 18:11:36.215977 2446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:36.217231 2306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:36.222018 2445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2456837bc6407e9e32d5503b0f4870. Latest consensus state: current_term: 1 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:36.222744 2445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:36.236910 2460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:11:36.244200 2460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:11:36.245180 2306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415: No bootstrap required, opened a new log
I20260925 18:11:36.253574 2306 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:36.256156 2306 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:36.256866 2306 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2641b1ebada4ee38658e3bf1fa31415, State: Initialized, Role: FOLLOWER
I20260925 18:11:36.258487 2306 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:36.263072 2412 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 1 FOLLOWER]: Refusing update from remote peer 9d2456837bc6407e9e32d5503b0f4870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 18:11:36.264572 2445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 18:11:36.267405 2469 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:36.268172 2469 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:36.270474 2306 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:11:36.270722 2279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:36.273185 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415"
I20260925 18:11:36.275332 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415" is_pre_election: true
I20260925 18:11:36.283813 2279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 1 FOLLOWER]: Refusing update from remote peer 9d2456837bc6407e9e32d5503b0f4870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 18:11:36.285286 2445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:36.289407 2444 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2456837bc6407e9e32d5503b0f4870. Latest consensus state: current_term: 1 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:36.290184 2444 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:36.294961 2466 mvcc.cc:205] Tried to move back new op lower bound from 7333314135076913152 to 7333314134908923904. Current Snapshot: MvccSnapshot[applied={T|T < 7333314135076913152}]
I20260925 18:11:36.309878 2444 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:36.310724 2444 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:36.314033 2445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:36.314783 2445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:36.315485 2460 catalog_manager.cc:1347] Generated new cluster ID: c1bce0eedc5e42e197d53912a6b6702a
I20260925 18:11:36.315755 2460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:11:36.322412 2477 catalog_manager.cc:1259] Loaded cluster ID: c1bce0eedc5e42e197d53912a6b6702a
I20260925 18:11:36.323091 2477 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: loading cluster ID for follower catalog manager: success
W20260925 18:11:36.327435 2477 catalog_manager.cc:1576] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260925 18:11:36.328012 2477 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260925 18:11:36.330495 2469 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2456837bc6407e9e32d5503b0f4870. Latest consensus state: current_term: 1 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:36.331262 2469 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:36.332860 2446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:36.333611 2446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:36.345149 2469 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:36.345957 2469 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:36.352584 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20260925 18:11:36.352878 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20260925 18:11:36.354564 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 18:11:36.357805 2492 catalog_manager.cc:1259] Loaded cluster ID: c1bce0eedc5e42e197d53912a6b6702a
I20260925 18:11:36.358054 2492 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415: loading cluster ID for follower catalog manager: success
W20260925 18:11:36.361702 2493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:36.362334 2492 catalog_manager.cc:1576] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260925 18:11:36.362761 2492 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20260925 18:11:36.367503 2494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:36.369946 2496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:36.370849 822 server_base.cc:1034] running on GCE node
I20260925 18:11:36.371665 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:36.371843 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:36.372004 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359896371990 us; error 0 us; skew 500 ppm
I20260925 18:11:36.372545 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:36.374941 822 webserver.cc:460] Webserver started at http://127.0.205.129:44345/ using document root <none> and password file <none>
I20260925 18:11:36.375386 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:36.375545 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:36.375828 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:36.376991 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "6ad91c285f824fc59f71f0e705d419a7"
format_stamp: "Formatted at 2026-09-25 18:11:36 on dist-test-slave-57m3"
I20260925 18:11:36.381956 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 18:11:36.385133 2501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:36.385859 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:11:36.386127 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "6ad91c285f824fc59f71f0e705d419a7"
format_stamp: "Formatted at 2026-09-25 18:11:36 on dist-test-slave-57m3"
I20260925 18:11:36.386394 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:36.406257 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:36.406494 2460 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:11:36.408077 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:36.408615 2460 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:11:36.410382 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:36.413677 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:36.413843 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:36.414094 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:36.414237 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:36.446588 2460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870: Generated new TSK 0
I20260925 18:11:36.447371 2460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:11:36.469444 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:46227
I20260925 18:11:36.469473 2563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:46227 every 8 connection(s)
I20260925 18:11:36.476032 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:36.494313 2570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:36.496598 2564 heartbeater.cc:347] Connected to a master server at 127.0.205.188:44299
I20260925 18:11:36.497002 2564 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:36.497797 2564 heartbeater.cc:511] Master 127.0.205.188:44299 requested a full tablet report, sending...
I20260925 18:11:36.500775 2402 ts_manager.cc:194] Registered new tserver with Master: 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
W20260925 18:11:36.508360 2572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:36.509133 2566 heartbeater.cc:347] Connected to a master server at 127.0.205.190:35227
I20260925 18:11:36.509531 2566 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:36.509527 2565 heartbeater.cc:347] Connected to a master server at 127.0.205.189:37349
I20260925 18:11:36.510056 2565 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:36.510346 2566 heartbeater.cc:511] Master 127.0.205.190:35227 requested a full tablet report, sending...
I20260925 18:11:36.510777 822 server_base.cc:1034] running on GCE node
I20260925 18:11:36.510926 2565 heartbeater.cc:511] Master 127.0.205.189:37349 requested a full tablet report, sending...
W20260925 18:11:36.511071 2574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:36.512020 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:36.512264 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:36.512460 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359896512445 us; error 0 us; skew 500 ppm
I20260925 18:11:36.513016 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:36.513608 2336 ts_manager.cc:194] Registered new tserver with Master: 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:11:36.514011 2271 ts_manager.cc:194] Registered new tserver with Master: 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:11:36.515753 2336 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52854
I20260925 18:11:36.518100 822 webserver.cc:460] Webserver started at http://127.0.205.130:42431/ using document root <none> and password file <none>
I20260925 18:11:36.518558 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:36.518729 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:36.518956 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:36.519888 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-1-root/instance:
uuid: "9f27b2d63b304859af2501907270d673"
format_stamp: "Formatted at 2026-09-25 18:11:36 on dist-test-slave-57m3"
I20260925 18:11:36.524026 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 18:11:36.527153 2579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:36.527937 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 18:11:36.528245 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-1-root
uuid: "9f27b2d63b304859af2501907270d673"
format_stamp: "Formatted at 2026-09-25 18:11:36 on dist-test-slave-57m3"
I20260925 18:11:36.528512 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:36.569012 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:36.570196 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:36.571372 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:36.573344 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:36.573511 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:36.573721 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:36.573856 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:36.618561 2565 heartbeater.cc:503] Master 127.0.205.189:37349 was elected leader, sending a full tablet report...
I20260925 18:11:36.654001 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.130:43505
I20260925 18:11:36.654107 2641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.130:43505 every 8 connection(s)
I20260925 18:11:36.666627 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 18:11:36.680154 2642 heartbeater.cc:347] Connected to a master server at 127.0.205.188:44299
I20260925 18:11:36.680599 2642 heartbeater.cc:464] Registering TS with master...
W20260925 18:11:36.681265 2649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:36.681504 2642 heartbeater.cc:511] Master 127.0.205.188:44299 requested a full tablet report, sending...
I20260925 18:11:36.684621 2402 ts_manager.cc:194] Registered new tserver with Master: 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:36.685879 2644 heartbeater.cc:347] Connected to a master server at 127.0.205.190:35227
I20260925 18:11:36.686745 2644 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:36.687758 2644 heartbeater.cc:511] Master 127.0.205.190:35227 requested a full tablet report, sending...
I20260925 18:11:36.690805 2271 ts_manager.cc:194] Registered new tserver with Master: 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:36.692031 822 server_base.cc:1034] running on GCE node
W20260925 18:11:36.692939 2650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:36.693364 2643 heartbeater.cc:347] Connected to a master server at 127.0.205.189:37349
I20260925 18:11:36.693746 2643 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:36.694466 2643 heartbeater.cc:511] Master 127.0.205.189:37349 requested a full tablet report, sending...
W20260925 18:11:36.695014 2652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:36.695861 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:36.696043 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:36.696198 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359896696186 us; error 0 us; skew 500 ppm
I20260925 18:11:36.696480 2336 ts_manager.cc:194] Registered new tserver with Master: 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:36.696817 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:36.698791 2336 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52858
I20260925 18:11:36.699316 822 webserver.cc:460] Webserver started at http://127.0.205.131:40809/ using document root <none> and password file <none>
I20260925 18:11:36.699826 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:36.700050 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:11:36.700295 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:11:36.701416 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-2-root/instance:
uuid: "74156d4911f84a01b82ea7651f6f13c6"
format_stamp: "Formatted at 2026-09-25 18:11:36 on dist-test-slave-57m3"
I20260925 18:11:36.705792 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 18:11:36.709508 2657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:36.710188 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:11:36.710435 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-2-root
uuid: "74156d4911f84a01b82ea7651f6f13c6"
format_stamp: "Formatted at 2026-09-25 18:11:36 on dist-test-slave-57m3"
I20260925 18:11:36.710695 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:36.731521 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:36.732955 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:36.735219 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:11:36.737987 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:11:36.738175 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:36.738418 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:11:36.738557 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:36.801748 2643 heartbeater.cc:503] Master 127.0.205.189:37349 was elected leader, sending a full tablet report...
I20260925 18:11:36.818984 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.131:42203
I20260925 18:11:36.819067 2719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.131:42203 every 8 connection(s)
I20260925 18:11:36.845083 2721 heartbeater.cc:347] Connected to a master server at 127.0.205.189:37349
I20260925 18:11:36.845507 2721 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:36.846908 2721 heartbeater.cc:511] Master 127.0.205.189:37349 requested a full tablet report, sending...
I20260925 18:11:36.847059 2720 heartbeater.cc:347] Connected to a master server at 127.0.205.188:44299
I20260925 18:11:36.847394 2720 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:36.848162 2720 heartbeater.cc:511] Master 127.0.205.188:44299 requested a full tablet report, sending...
I20260925 18:11:36.848968 2336 ts_manager.cc:194] Registered new tserver with Master: 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:36.850613 2402 ts_manager.cc:194] Registered new tserver with Master: 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:36.850652 2722 heartbeater.cc:347] Connected to a master server at 127.0.205.190:35227
I20260925 18:11:36.850849 2336 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52868
I20260925 18:11:36.851207 2722 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:36.852052 2722 heartbeater.cc:511] Master 127.0.205.190:35227 requested a full tablet report, sending...
I20260925 18:11:36.854264 2271 ts_manager.cc:194] Registered new tserver with Master: 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:36.855284 822 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.024851438s
I20260925 18:11:36.903811 2336 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20260925 18:11:36.905400 2336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 18:11:36.954879 2721 heartbeater.cc:503] Master 127.0.205.189:37349 was elected leader, sending a full tablet report...
I20260925 18:11:36.987330 2685 tablet_service.cc:1467] Processing CreateTablet for tablet bd005713f7db4ce8aca0658034f5b7cb (DEFAULT_TABLE table=test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:36.987505 2684 tablet_service.cc:1467] Processing CreateTablet for tablet df8c60f75ea94608b6072d8e17e3e356 (DEFAULT_TABLE table=test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:36.988880 2685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd005713f7db4ce8aca0658034f5b7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:36.990785 2684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8c60f75ea94608b6072d8e17e3e356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:36.990921 2683 tablet_service.cc:1467] Processing CreateTablet for tablet ab34f9c8e4e74a4e981efa7270e886da (DEFAULT_TABLE table=test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:36.992509 2683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab34f9c8e4e74a4e981efa7270e886da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:36.999008 2607 tablet_service.cc:1467] Processing CreateTablet for tablet bd005713f7db4ce8aca0658034f5b7cb (DEFAULT_TABLE table=test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:37.000010 2605 tablet_service.cc:1467] Processing CreateTablet for tablet ab34f9c8e4e74a4e981efa7270e886da (DEFAULT_TABLE table=test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:37.000937 2607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd005713f7db4ce8aca0658034f5b7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:37.001600 2605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab34f9c8e4e74a4e981efa7270e886da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:37.002821 2606 tablet_service.cc:1467] Processing CreateTablet for tablet df8c60f75ea94608b6072d8e17e3e356 (DEFAULT_TABLE table=test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:37.004526 2606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8c60f75ea94608b6072d8e17e3e356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:37.016781 2529 tablet_service.cc:1467] Processing CreateTablet for tablet bd005713f7db4ce8aca0658034f5b7cb (DEFAULT_TABLE table=test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:37.018663 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd005713f7db4ce8aca0658034f5b7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:37.018836 2741 tablet_bootstrap.cc:492] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6: Bootstrap starting.
I20260925 18:11:37.021974 2528 tablet_service.cc:1467] Processing CreateTablet for tablet df8c60f75ea94608b6072d8e17e3e356 (DEFAULT_TABLE table=test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:37.023519 2528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df8c60f75ea94608b6072d8e17e3e356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:37.026245 2527 tablet_service.cc:1467] Processing CreateTablet for tablet ab34f9c8e4e74a4e981efa7270e886da (DEFAULT_TABLE table=test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260925 18:11:37.027534 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab34f9c8e4e74a4e981efa7270e886da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:11:37.027952 2741 tablet_bootstrap.cc:654] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:37.080101 2744 tablet_bootstrap.cc:492] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7: Bootstrap starting.
I20260925 18:11:37.088260 2742 tablet_bootstrap.cc:492] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673: Bootstrap starting.
I20260925 18:11:37.088966 2744 tablet_bootstrap.cc:654] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:37.091529 2741 tablet_bootstrap.cc:492] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6: No bootstrap required, opened a new log
I20260925 18:11:37.092089 2741 ts_tablet_manager.cc:1397] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6: Time spent bootstrapping tablet: real 0.074s user 0.020s sys 0.017s
I20260925 18:11:37.096585 2744 tablet_bootstrap.cc:492] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7: No bootstrap required, opened a new log
I20260925 18:11:37.096697 2742 tablet_bootstrap.cc:654] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:37.097119 2744 ts_tablet_manager.cc:1397] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260925 18:11:37.094986 2741 raft_consensus.cc:348] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.098543 2741 raft_consensus.cc:374] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:37.098866 2741 raft_consensus.cc:729] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Initialized, Role: FOLLOWER
I20260925 18:11:37.099596 2741 consensus_queue.cc:260] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.100205 2744 raft_consensus.cc:348] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.100967 2744 raft_consensus.cc:374] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:37.101279 2744 raft_consensus.cc:729] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Initialized, Role: FOLLOWER
I20260925 18:11:37.101997 2744 consensus_queue.cc:260] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.107357 2744 ts_tablet_manager.cc:1428] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260925 18:11:37.108294 2744 tablet_bootstrap.cc:492] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7: Bootstrap starting.
I20260925 18:11:37.116108 2744 tablet_bootstrap.cc:654] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:37.119796 2742 tablet_bootstrap.cc:492] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673: No bootstrap required, opened a new log
I20260925 18:11:37.120355 2742 ts_tablet_manager.cc:1397] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.014s
I20260925 18:11:37.122889 2744 tablet_bootstrap.cc:492] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7: No bootstrap required, opened a new log
I20260925 18:11:37.123478 2744 ts_tablet_manager.cc:1397] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260925 18:11:37.122905 2742 raft_consensus.cc:348] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.123853 2742 raft_consensus.cc:374] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:37.124147 2742 raft_consensus.cc:729] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f27b2d63b304859af2501907270d673, State: Initialized, Role: FOLLOWER
I20260925 18:11:37.133881 2741 ts_tablet_manager.cc:1428] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6: Time spent starting tablet: real 0.041s user 0.008s sys 0.008s
I20260925 18:11:37.134444 2744 raft_consensus.cc:348] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:37.135270 2744 raft_consensus.cc:374] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:37.135646 2744 raft_consensus.cc:729] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Initialized, Role: FOLLOWER
I20260925 18:11:37.136337 2744 consensus_queue.cc:260] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:37.140036 2741 tablet_bootstrap.cc:492] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6: Bootstrap starting.
I20260925 18:11:37.139056 2744 ts_tablet_manager.cc:1428] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260925 18:11:37.144716 2744 tablet_bootstrap.cc:492] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7: Bootstrap starting.
I20260925 18:11:37.149511 2747 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:37.151477 2747 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.154796 2742 consensus_queue.cc:260] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.165491 2744 tablet_bootstrap.cc:654] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:37.166097 2750 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:37.158452 2747 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:37.175186 2741 tablet_bootstrap.cc:654] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:37.167402 2750 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.190805 2746 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:37.191649 2746 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.202819 2750 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:37.211265 2744 tablet_bootstrap.cc:492] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7: No bootstrap required, opened a new log
I20260925 18:11:37.211755 2744 ts_tablet_manager.cc:1397] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7: Time spent bootstrapping tablet: real 0.067s user 0.023s sys 0.012s
I20260925 18:11:37.214267 2744 raft_consensus.cc:348] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.214988 2744 raft_consensus.cc:374] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:37.215312 2744 raft_consensus.cc:729] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Initialized, Role: FOLLOWER
I20260925 18:11:37.218014 2747 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:37.218492 2747 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:37.217151 2746 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:37.220853 2744 consensus_queue.cc:260] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.245091 2747 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:37.248428 2742 ts_tablet_manager.cc:1428] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673: Time spent starting tablet: real 0.128s user 0.009s sys 0.029s
I20260925 18:11:37.249023 2747 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:37.249526 2747 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.250105 2742 tablet_bootstrap.cc:492] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673: Bootstrap starting.
I20260925 18:11:37.258574 2747 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:37.260478 2741 tablet_bootstrap.cc:492] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6: No bootstrap required, opened a new log
I20260925 18:11:37.261209 2741 ts_tablet_manager.cc:1397] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6: Time spent bootstrapping tablet: real 0.121s user 0.012s sys 0.014s
I20260925 18:11:37.267666 2741 raft_consensus.cc:348] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:37.268682 2741 raft_consensus.cc:374] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:37.269307 2741 raft_consensus.cc:729] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Initialized, Role: FOLLOWER
I20260925 18:11:37.270267 2741 consensus_queue.cc:260] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:37.285174 2741 ts_tablet_manager.cc:1428] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6: Time spent starting tablet: real 0.024s user 0.005s sys 0.003s
I20260925 18:11:37.292264 2744 ts_tablet_manager.cc:1428] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7: Time spent starting tablet: real 0.080s user 0.012s sys 0.007s
I20260925 18:11:37.291807 2741 tablet_bootstrap.cc:492] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6: Bootstrap starting.
I20260925 18:11:37.301519 2742 tablet_bootstrap.cc:654] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:37.303982 2746 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:37.308296 2746 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:37.314780 2746 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:11:37.332576 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:37.330068 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:37.329456 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:37.333657 2615 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 0.
W20260925 18:11:37.338944 2502 leader_election.cc:341] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260925 18:11:37.339632 2502 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters:
W20260925 18:11:37.340559 2502 leader_election.cc:341] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260925 18:11:37.340873 2741 tablet_bootstrap.cc:654] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:37.342900 2759 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 18:11:37.352200 2759 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:11:37.352602 2759 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:37.361963 2742 tablet_bootstrap.cc:492] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673: No bootstrap required, opened a new log
I20260925 18:11:37.362932 2742 ts_tablet_manager.cc:1397] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673: Time spent bootstrapping tablet: real 0.113s user 0.010s sys 0.002s
I20260925 18:11:37.363632 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:37.364423 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:37.369947 2694 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 0.
I20260925 18:11:37.370944 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:37.373021 2693 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 0.
I20260925 18:11:37.375972 2742 raft_consensus.cc:348] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.377130 2742 raft_consensus.cc:374] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:37.377461 2742 raft_consensus.cc:729] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f27b2d63b304859af2501907270d673, State: Initialized, Role: FOLLOWER
I20260925 18:11:37.382771 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:37.378422 2742 consensus_queue.cc:260] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.385345 2741 tablet_bootstrap.cc:492] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6: No bootstrap required, opened a new log
I20260925 18:11:37.386063 2759 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.386845 2741 ts_tablet_manager.cc:1397] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6: Time spent bootstrapping tablet: real 0.098s user 0.011s sys 0.002s
I20260925 18:11:37.386801 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:37.383384 2693 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 0.
I20260925 18:11:37.389227 2539 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 in current term 1: Already voted for candidate 6ad91c285f824fc59f71f0e705d419a7 in this term.
I20260925 18:11:37.390393 2759 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 election: Requested vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:37.390625 2742 ts_tablet_manager.cc:1428] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673: Time spent starting tablet: real 0.027s user 0.009s sys 0.000s
I20260925 18:11:37.391140 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:37.391680 2617 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:37.392061 2742 tablet_bootstrap.cc:492] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673: Bootstrap starting.
I20260925 18:11:37.393503 2477 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: acquiring CA information for follower catalog manager: success
I20260925 18:11:37.397485 2581 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:11:37.397591 2503 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters: 9f27b2d63b304859af2501907270d673
I20260925 18:11:37.403301 2741 raft_consensus.cc:348] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.406235 2741 raft_consensus.cc:374] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:37.406541 2741 raft_consensus.cc:729] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Initialized, Role: FOLLOWER
I20260925 18:11:37.407299 2741 consensus_queue.cc:260] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.408808 2742 tablet_bootstrap.cc:654] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673: Neither blocks nor log segments found. Creating new log.
I20260925 18:11:37.419662 2759 raft_consensus.cc:2781] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 18:11:37.420177 2759 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:11:37.420591 2759 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:37.423195 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:37.423791 2693 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:37.424180 2477 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 18:11:37.424429 2746 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:37.424911 2746 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.432585 2746 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:37.423207 2492 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415: acquiring CA information for follower catalog manager: success
I20260925 18:11:37.438910 2741 ts_tablet_manager.cc:1428] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6: Time spent starting tablet: real 0.051s user 0.003s sys 0.004s
I20260925 18:11:37.438920 2693 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 1.
I20260925 18:11:37.439957 2617 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 1.
I20260925 18:11:37.440434 2754 raft_consensus.cc:2741] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260925 18:11:37.447177 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:37.447484 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:37.447927 2617 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74156d4911f84a01b82ea7651f6f13c6 in current term 1: Already voted for candidate 6ad91c285f824fc59f71f0e705d419a7 in this term.
I20260925 18:11:37.448822 2616 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 0.
I20260925 18:11:37.446389 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:37.446544 2502 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:11:37.451738 2658 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:11:37.452534 2746 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260925 18:11:37.452651 2658 leader_election.cc:341] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260925 18:11:37.452931 2746 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:11:37.453301 2746 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:37.455482 2742 tablet_bootstrap.cc:492] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673: No bootstrap required, opened a new log
I20260925 18:11:37.455983 2742 ts_tablet_manager.cc:1397] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.013s
I20260925 18:11:37.457506 2761 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:11:37.457983 2761 raft_consensus.cc:686] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 1 LEADER]: Becoming Leader. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
I20260925 18:11:37.458503 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:37.458797 2742 raft_consensus.cc:348] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:37.459388 2742 raft_consensus.cc:374] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:11:37.459183 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:37.459632 2742 raft_consensus.cc:729] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f27b2d63b304859af2501907270d673, State: Initialized, Role: FOLLOWER
I20260925 18:11:37.460228 2742 consensus_queue.cc:260] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:37.458738 2761 consensus_queue.cc:237] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.461614 2746 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.464488 2660 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:11:37.465468 2746 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 1 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:37.466058 2746 raft_consensus.cc:2726] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 18:11:37.467283 2742 ts_tablet_manager.cc:1428] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20260925 18:11:37.469790 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:11:37.470374 2539 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:37.471457 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:37.473102 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:37.473680 2615 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:37.478247 2539 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 1.
I20260925 18:11:37.478591 2615 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 1.
I20260925 18:11:37.479377 2660 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters:
I20260925 18:11:37.480527 2767 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:11:37.481033 2767 raft_consensus.cc:686] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 1 LEADER]: Becoming Leader. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:11:37.481890 2759 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:37.482945 2538 raft_consensus.cc:2379] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74156d4911f84a01b82ea7651f6f13c6 in current term 1: Already voted for candidate 6ad91c285f824fc59f71f0e705d419a7 in this term.
I20260925 18:11:37.482757 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:37.481772 2767 consensus_queue.cc:237] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:37.484983 2660 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:11:37.485476 2503 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7; no voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673
I20260925 18:11:37.485985 2746 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:37.486645 2747 raft_consensus.cc:2726] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 18:11:37.487100 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:37.487818 2615 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:11:37.492043 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:37.492795 2746 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 18:11:37.493264 2759 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 election: Requested vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:37.499930 2615 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 1.
I20260925 18:11:37.487836 2492 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 18:11:37.501044 2502 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:11:37.503713 2336 catalog_manager.cc:5495] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129). New cstate: current_term: 1 leader_uuid: "6ad91c285f824fc59f71f0e705d419a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20260925 18:11:37.505667 2761 raft_consensus.cc:2781] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:11:37.506175 2761 raft_consensus.cc:686] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 1 LEADER]: Becoming Leader. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
I20260925 18:11:37.507035 2761 consensus_queue.cc:237] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:37.511718 2335 catalog_manager.cc:5495] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131). New cstate: current_term: 1 leader_uuid: "74156d4911f84a01b82ea7651f6f13c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: HEALTHY } } }
I20260925 18:11:37.517241 2693 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 1.
I20260925 18:11:37.541529 2336 catalog_manager.cc:5495] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129). New cstate: current_term: 1 leader_uuid: "6ad91c285f824fc59f71f0e705d419a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: HEALTHY } } }
I20260925 18:11:37.558001 2761 consensus_queue.cc:1035] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:37.576607 2747 consensus_queue.cc:1035] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:37.591926 2746 consensus_queue.cc:1035] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:37.594916 2759 consensus_queue.cc:1035] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260925 18:11:37.616954 2761 consensus_queue.cc:1035] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:37.629009 2767 consensus_queue.cc:1035] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:11:38.250378 2538 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da"
dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
from {username='slave'} at 127.0.0.1:47318
I20260925 18:11:38.251044 2538 raft_consensus.cc:580] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 1 LEADER]: Received request to step down
I20260925 18:11:38.251350 2538 raft_consensus.cc:3032] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 18:11:38.251632 2538 raft_consensus.cc:729] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
I20260925 18:11:38.252213 2538 consensus_queue.cc:260] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.253759 2538 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:38.391353 2746 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:38.391819 2746 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.393406 2746 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:38.394161 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:38.394323 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:38.394656 2538 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 2.
I20260925 18:11:38.395855 2660 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters: 9f27b2d63b304859af2501907270d673
I20260925 18:11:38.396557 2746 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 18:11:38.396778 2808 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:38.396881 2746 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:38.397327 2746 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:38.397308 2808 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.398995 2808 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:38.399775 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:38.400254 2538 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 2.
I20260925 18:11:38.400017 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:38.401166 2582 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:11:38.401801 2808 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 18:11:38.402057 2808 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:38.402307 2808 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:38.402669 2746 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.403609 2695 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 in current term 2: Already voted for candidate 74156d4911f84a01b82ea7651f6f13c6 in this term.
I20260925 18:11:38.405292 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:38.405901 2746 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 2 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:38.406937 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:11:38.408141 2808 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.409961 2808 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 2 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:38.410851 2617 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 74156d4911f84a01b82ea7651f6f13c6 in current term 2: Already voted for candidate 9f27b2d63b304859af2501907270d673 in this term.
I20260925 18:11:38.414068 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:38.414937 2695 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 in current term 2: Already voted for candidate 74156d4911f84a01b82ea7651f6f13c6 in this term.
I20260925 18:11:38.416635 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:11:38.417183 2538 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 2.
I20260925 18:11:38.418073 2582 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:11:38.419500 2660 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters: 9f27b2d63b304859af2501907270d673
I20260925 18:11:38.420285 2746 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 18:11:38.420851 2746 raft_consensus.cc:686] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 2 LEADER]: Becoming Leader. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:11:38.422284 2808 raft_consensus.cc:2726] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 18:11:38.423225 2746 consensus_queue.cc:237] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.432060 2335 catalog_manager.cc:5495] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 reported cstate change: term changed from 1 to 2, leader changed from 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129) to 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131). New cstate: current_term: 2 leader_uuid: "74156d4911f84a01b82ea7651f6f13c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: HEALTHY } } }
I20260925 18:11:38.471282 2538 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb"
dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
from {username='slave'} at 127.0.0.1:47318
I20260925 18:11:38.471858 2538 raft_consensus.cc:580] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 1 LEADER]: Received request to step down
I20260925 18:11:38.472138 2538 raft_consensus.cc:3032] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 18:11:38.472378 2538 raft_consensus.cc:729] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
I20260925 18:11:38.472966 2538 consensus_queue.cc:260] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:38.473798 2538 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:38.516800 2538 raft_consensus.cc:1261] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Refusing update from remote peer 74156d4911f84a01b82ea7651f6f13c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 18:11:38.518064 2746 consensus_queue.cc:1035] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:11:38.518813 2617 raft_consensus.cc:1261] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Refusing update from remote peer 74156d4911f84a01b82ea7651f6f13c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 18:11:38.520002 2746 consensus_queue.cc:1035] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:11:38.559561 2785 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:38.560079 2785 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:38.561960 2785 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:11:38.562597 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:38.563180 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:38.563658 2538 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 2.
I20260925 18:11:38.564328 2660 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters: 9f27b2d63b304859af2501907270d673
I20260925 18:11:38.564903 2785 raft_consensus.cc:2781] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 18:11:38.565168 2785 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:38.565382 2785 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:38.570699 2785 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:38.572310 2785 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 2 election: Requested vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:11:38.573041 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:38.573626 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:11:38.578034 2538 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 2.
I20260925 18:11:38.578780 2660 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters: 9f27b2d63b304859af2501907270d673
I20260925 18:11:38.579380 2785 raft_consensus.cc:2781] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 18:11:38.579728 2785 raft_consensus.cc:686] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 2 LEADER]: Becoming Leader. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:11:38.580418 2785 consensus_queue.cc:237] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:38.587074 2333 catalog_manager.cc:5495] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 reported cstate change: term changed from 1 to 2, leader changed from 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129) to 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131). New cstate: current_term: 2 leader_uuid: "74156d4911f84a01b82ea7651f6f13c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: UNKNOWN } } }
I20260925 18:11:38.609318 2808 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:38.609750 2808 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:38.611261 2808 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:11:38.612388 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:38.612465 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:38.613121 2538 raft_consensus.cc:2379] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 in current term 2: Already voted for candidate 74156d4911f84a01b82ea7651f6f13c6 in this term.
I20260925 18:11:38.613873 2582 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:11:38.614472 2808 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:38.619583 2808 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 18:11:38.662683 2538 raft_consensus.cc:1261] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Refusing update from remote peer 74156d4911f84a01b82ea7651f6f13c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 18:11:38.663856 2785 consensus_queue.cc:1035] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:11:38.672564 2617 raft_consensus.cc:1261] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Refusing update from remote peer 74156d4911f84a01b82ea7651f6f13c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 18:11:38.677511 2785 consensus_queue.cc:1035] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:11:38.707270 2695 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356"
dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
from {username='slave'} at 127.0.0.1:48936
I20260925 18:11:38.707875 2695 raft_consensus.cc:580] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 1 LEADER]: Received request to step down
I20260925 18:11:38.708200 2695 raft_consensus.cc:3032] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 18:11:38.708477 2695 raft_consensus.cc:729] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:11:38.709194 2695 consensus_queue.cc:260] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.710397 2695 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:38.852692 2761 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:38.853225 2761 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.854629 2761 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:38.855468 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:38.855525 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:38.856125 2695 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 2.
I20260925 18:11:38.857010 2503 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters: 9f27b2d63b304859af2501907270d673
I20260925 18:11:38.857741 2761 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 18:11:38.858078 2761 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:38.858400 2761 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:38.862969 2761 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.864400 2761 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 2 election: Requested vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:38.865073 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:38.865291 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:38.869755 2695 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 2.
I20260925 18:11:38.870639 2503 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters: 9f27b2d63b304859af2501907270d673
I20260925 18:11:38.871438 2761 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 18:11:38.872303 2761 raft_consensus.cc:686] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 2 LEADER]: Becoming Leader. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
I20260925 18:11:38.873031 2761 consensus_queue.cc:237] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.880919 2335 catalog_manager.cc:5495] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 reported cstate change: term changed from 1 to 2, leader changed from 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131) to 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129). New cstate: current_term: 2 leader_uuid: "6ad91c285f824fc59f71f0e705d419a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20260925 18:11:38.895524 2808 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:38.896167 2808 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:38.897912 2808 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:38.898854 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:38.899473 2695 raft_consensus.cc:2379] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 in current term 2: Already voted for candidate 6ad91c285f824fc59f71f0e705d419a7 in this term.
I20260925 18:11:38.900283 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:38.902118 2582 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:11:38.902734 2808 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:38.906792 2808 raft_consensus.cc:2726] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 18:11:38.916270 2695 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb"
dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
from {username='slave'} at 127.0.0.1:48936
I20260925 18:11:38.916903 2695 raft_consensus.cc:580] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 2 LEADER]: Received request to step down
I20260925 18:11:38.917235 2695 raft_consensus.cc:3032] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 18:11:38.917508 2695 raft_consensus.cc:729] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:11:38.918704 2695 consensus_queue.cc:260] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:38.919852 2695 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:38.954604 2617 raft_consensus.cc:1261] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Refusing update from remote peer 6ad91c285f824fc59f71f0e705d419a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 18:11:38.955660 2761 consensus_queue.cc:1035] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:11:38.969300 2695 raft_consensus.cc:1261] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Refusing update from remote peer 6ad91c285f824fc59f71f0e705d419a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 18:11:38.970757 2820 consensus_queue.cc:1035] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:11:39.059288 2761 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:39.059950 2761 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:39.061714 2808 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:39.063659 2761 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:39.062347 2808 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:39.362576 2808 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
W20260925 18:11:41.145358 2433 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1280 after lost signal to thread 825
I20260925 18:11:41.450358 2824 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9d2456837bc6407e9e32d5503b0f4870)
I20260925 18:11:41.450982 2824 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:41.454007 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415" is_pre_election: true
I20260925 18:11:41.454713 2281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 1.
I20260925 18:11:41.455912 2385 leader_election.cc:302] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2641b1ebada4ee38658e3bf1fa31415, fc858f182cf448a3a94ba08cb41d69c0; no voters:
I20260925 18:11:41.462090 2825 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:41.462666 2825 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:41.481127 2824 leader_election.cc:288] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349)
I20260925 18:11:41.464769 2825 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:41.484876 2824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 18:11:41.486155 2824 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9d2456837bc6407e9e32d5503b0f4870)
I20260925 18:11:41.486519 2824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:41.488566 2827 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9d2456837bc6407e9e32d5503b0f4870)
I20260925 18:11:41.489112 2827 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:41.496568 2827 leader_election.cc:288] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:11:41.531946 2824 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:41.533955 2824 leader_election.cc:288] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 2 election: Requested vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349)
W20260925 18:11:42.056200 2256 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.000s sys 0.015s
W20260925 18:11:42.057329 2256 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.000s sys 0.016s
W20260925 18:11:42.063169 2385 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.205.189:37349 timed out after 0.271s (SENT) [suppressed 6 similar messages]
I20260925 18:11:42.068756 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415"
I20260925 18:11:42.068471 2344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870"
I20260925 18:11:42.069518 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:11:42.070235 2281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 1 FOLLOWER]: Advancing to term 2
W20260925 18:11:42.115108 2385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.189:37349, user_credentials={real_user=slave}} blocked reactor thread for 52129.5us
I20260925 18:11:42.147769 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:11:42.319482 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415"
I20260925 18:11:42.321425 2385 leader_election.cc:302] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc858f182cf448a3a94ba08cb41d69c0; no voters: 9d2456837bc6407e9e32d5503b0f4870, c2641b1ebada4ee38658e3bf1fa31415
I20260925 18:11:42.481741 2411 raft_consensus.cc:1226] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 2 FOLLOWER]: Rejecting Update request from peer 9d2456837bc6407e9e32d5503b0f4870 for earlier term 1. Current term is 2. Ops: []
I20260925 18:11:42.482638 2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44120 (request call id 50) took 1139 ms. Trace:
I20260925 18:11:42.483110 2411 rpcz_store.cc:276] 0925 18:11:41.342943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:11:42.480898 (+1137955us) service_pool.cc:227] Handling call
0925 18:11:42.481411 (+ 513us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 18:11:42.482312 (+ 901us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 18:11:42.482538 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 18:11:42.495743 2281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 2.
W20260925 18:11:42.568598 2281 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60452 (request call id 2) took 501 ms (client timeout 213 ms). Trace:
W20260925 18:11:42.569134 2281 rpcz_store.cc:269] 0925 18:11:42.067807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:11:42.068198 (+ 391us) service_pool.cc:227] Handling call
0925 18:11:42.568494 (+500296us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 18:11:42.544762 2255 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.189:37349, user_credentials={real_user=slave}} blocked reactor thread for 246812us
I20260925 18:11:42.572285 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:11:42.575253 2834 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:42.575987 2834 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:42.578370 2834 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:42.567226 2833 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 18:11:42.562110 2582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.129:46227, user_credentials={real_user=slave}} blocked reactor thread for 490521us
I20260925 18:11:42.608271 2695 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da"
dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
from {username='slave'} at 127.0.0.1:48936
I20260925 18:11:42.609776 2695 raft_consensus.cc:580] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 2 LEADER]: Received request to step down
I20260925 18:11:42.610179 2695 raft_consensus.cc:3032] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 18:11:42.579643 2832 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 18:11:42.580379 2385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.190:35227, user_credentials={real_user=slave}} blocked reactor thread for 259165us
I20260925 18:11:42.610461 2695 raft_consensus.cc:729] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:11:42.596022 2411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0" is_pre_election: true
I20260925 18:11:42.597244 2538 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47306 (request call id 34) took 3535 ms. Trace:
W20260925 18:11:42.601765 2346 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52836 (request call id 1) took 542 ms (client timeout 240 ms). Trace:
I20260925 18:11:42.613370 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0" is_pre_election: true
I20260925 18:11:42.611984 2695 consensus_queue.cc:260] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:42.612442 2538 rpcz_store.cc:276] 0925 18:11:42.595776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:11:42.596029 (+ 253us) service_pool.cc:227] Handling call
0925 18:11:42.596456 (+ 427us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 18:11:42.596706 (+ 250us) raft_consensus.cc:1520] Early marking committed up to index 0
0925 18:11:42.596754 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0925 18:11:42.596817 (+ 63us) raft_consensus.cc:1631] Marking committed up to 3
0925 18:11:42.596869 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 18:11:42.596939 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0925 18:11:42.597047 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 18:11:42.615553 2411 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2641b1ebada4ee38658e3bf1fa31415 in current term 2: Already voted for candidate fc858f182cf448a3a94ba08cb41d69c0 in this term.
I20260925 18:11:42.616248 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:42.623386 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:42.627645 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:42.636353 2833 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:42.646317 2535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:42.647341 2255 leader_election.cc:302] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2641b1ebada4ee38658e3bf1fa31415; no voters: 9d2456837bc6407e9e32d5503b0f4870, fc858f182cf448a3a94ba08cb41d69c0
I20260925 18:11:42.646401 2539 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 2.
W20260925 18:11:42.623478 2346 rpcz_store.cc:269] 0925 18:11:42.059787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:11:42.601155 (+541368us) service_pool.cc:214] Skipping call since client already timed out
0925 18:11:42.601540 (+ 385us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 18:11:42.657541 2695 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:42.664304 2279 raft_consensus.cc:1226] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 2 FOLLOWER]: Rejecting Update request from peer 9d2456837bc6407e9e32d5503b0f4870 for earlier term 1. Current term is 2. Ops: []
I20260925 18:11:42.665143 2279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60442 (request call id 46) took 1251 ms. Trace:
I20260925 18:11:42.665496 2279 rpcz_store.cc:276] 0925 18:11:41.413408 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:11:42.660529 (+1247121us) service_pool.cc:227] Handling call
0925 18:11:42.663978 (+ 3449us) raft_consensus.cc:1457] Updating replica for 0 ops
0925 18:11:42.664783 (+ 805us) raft_consensus.cc:1683] Filling consensus response to leader.
0925 18:11:42.665036 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 18:11:42.662082 2484 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 }, Status: INVALID_TERM, Last received: 1.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.009s
W20260925 18:11:42.675354 2433 debug-util.cc:398] Leaking SignalData structure 0x7b0800166d40 after lost signal to thread 2305
W20260925 18:11:42.676625 2433 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc560 after lost signal to thread 2370
I20260925 18:11:42.681989 2833 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:11:42.684281 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:42.685132 2617 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 2.
I20260925 18:11:42.687276 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:42.688735 2693 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 3.
W20260925 18:11:42.689141 2433 debug-util.cc:398] Leaking SignalData structure 0x7b0800224b60 after lost signal to thread 2436
W20260925 18:11:42.690641 2433 debug-util.cc:398] Leaking SignalData structure 0x7b080013ae40 after lost signal to thread 2563
I20260925 18:11:42.690860 2503 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters:
W20260925 18:11:42.691654 2433 debug-util.cc:398] Leaking SignalData structure 0x7b0800237320 after lost signal to thread 2641
I20260925 18:11:42.692247 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:42.694281 2693 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 3.
I20260925 18:11:42.695182 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
W20260925 18:11:42.692662 2433 debug-util.cc:398] Leaking SignalData structure 0x7b08000e9da0 after lost signal to thread 2719
I20260925 18:11:42.704764 2836 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:42.705487 2836 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:42.706598 2616 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 2.
I20260925 18:11:42.707823 2836 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
W20260925 18:11:42.709448 2255 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.188:44299, user_credentials={real_user=slave}} blocked reactor thread for 62235.9us
I20260925 18:11:42.708591 2838 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:42.711340 2838 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:42.696066 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:42.713965 2839 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 18:11:42.714995 2838 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:42.718554 2841 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 1 LEADER]: Stepping down as leader of term 1
I20260925 18:11:42.723102 2841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9d2456837bc6407e9e32d5503b0f4870, State: Running, Role: LEADER
I20260925 18:11:42.719312 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:42.719661 2837 raft_consensus.cc:2781] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 18:11:42.724395 2837 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:42.724884 2837 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Advancing to term 3
W20260925 18:11:42.713534 2433 debug-util.cc:398] Leaking SignalData structure 0x7b080023f4e0 after lost signal to thread 2768
W20260925 18:11:42.725574 2433 debug-util.cc:398] Leaking SignalData structure 0x7b0800251da0 after lost signal to thread 2771
I20260925 18:11:42.725109 2841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
W20260925 18:11:42.725858 2433 debug-util.cc:398] Leaking SignalData structure 0x7b0800068c80 after lost signal to thread 2783
I20260925 18:11:42.726954 2484 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 }, Status: INVALID_TERM, Last received: 1.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260925 18:11:42.721940 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:42.738271 2694 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 3.
I20260925 18:11:42.722548 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:42.739562 2695 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 3.
I20260925 18:11:42.719086 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:42.740509 2615 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 2.
I20260925 18:11:42.723903 2693 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 2.
I20260925 18:11:42.742554 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:42.744763 2581 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:11:42.745666 2836 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 18:11:42.745883 2581 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7
I20260925 18:11:42.746047 2836 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:42.746963 2836 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:42.745128 2658 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:11:42.750432 2838 raft_consensus.cc:2781] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 18:11:42.750829 2838 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:11:42.751158 2838 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:11:42.753609 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
W20260925 18:11:42.754556 2503 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.131:42203, user_credentials={real_user=slave}} blocked reactor thread for 63930.4us
I20260925 18:11:42.753837 2502 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:11:42.755991 2840 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 18:11:42.756376 2840 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:42.756721 2840 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:42.758514 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:42.760126 2841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:11:42.760490 2844 raft_consensus.cc:2781] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 18:11:42.760494 2660 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:11:42.761324 2581 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7
I20260925 18:11:42.761601 2835 raft_consensus.cc:2726] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 18:11:42.761719 2844 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 74156d4911f84a01b82ea7651f6f13c6)
I20260925 18:11:42.762439 2844 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:42.763831 2845 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 18:11:42.764266 2845 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:42.764632 2845 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 2 FOLLOWER]: Advancing to term 3
W20260925 18:11:42.765085 2320 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.205.190:35227, user_credentials={real_user=slave}} blocked reactor thread for 97017.2us
I20260925 18:11:42.825937 2832 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:42.826453 2832 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:42.828493 2832 leader_election.cc:288] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349)
I20260925 18:11:42.829551 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415" is_pre_election: true
I20260925 18:11:42.830221 2279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 2.
I20260925 18:11:42.829577 2346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:11:42.831547 2385 leader_election.cc:302] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2641b1ebada4ee38658e3bf1fa31415, fc858f182cf448a3a94ba08cb41d69c0; no voters:
I20260925 18:11:42.832636 2832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 18:11:42.838501 2832 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:11:42.838829 2832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:42.914561 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:42.940640 2836 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:42.941737 2838 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:42.944770 2832 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:42.946696 2832 leader_election.cc:288] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 3 election: Requested vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349)
I20260925 18:11:42.948974 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870"
I20260925 18:11:42.949393 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415"
I20260925 18:11:42.950049 2279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:42.952291 2836 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:42.953092 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:11:42.940734 2840 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:42.956175 2537 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 in current term 3: Already voted for candidate 6ad91c285f824fc59f71f0e705d419a7 in this term.
I20260925 18:11:42.957108 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:42.957892 2838 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 election: Requested vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:11:42.958853 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:11:42.959322 2346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 2.
I20260925 18:11:42.960711 2840 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 3 election: Requested vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:42.961231 2484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:42.961668 2484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:42.963074 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:42.964948 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:42.965719 2615 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ad91c285f824fc59f71f0e705d419a7 in current term 3: Already voted for candidate 9f27b2d63b304859af2501907270d673 in this term.
I20260925 18:11:42.967056 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:42.967594 2502 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7; no voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673
I20260925 18:11:42.969172 2840 raft_consensus.cc:2726] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 18:11:42.970312 2484 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:11:42.971844 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0" is_pre_election: true
I20260925 18:11:42.973448 2412 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9d2456837bc6407e9e32d5503b0f4870 in current term 3: Already voted for candidate fc858f182cf448a3a94ba08cb41d69c0 in this term.
I20260925 18:11:42.975601 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415" is_pre_election: true
I20260925 18:11:42.941195 2837 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:42.981297 2320 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d2456837bc6407e9e32d5503b0f4870; no voters: c2641b1ebada4ee38658e3bf1fa31415, fc858f182cf448a3a94ba08cb41d69c0
I20260925 18:11:42.981379 2536 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 3.
W20260925 18:11:42.982615 2536 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47306 (request call id 35) took 288 ms (client timeout 226 ms). Trace:
I20260925 18:11:42.983080 2837 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 3 election: Requested vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:42.983923 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:42.984669 2695 raft_consensus.cc:2354] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ad91c285f824fc59f71f0e705d419a7 for earlier term 3. Current term is 4.
I20260925 18:11:42.985644 2484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:42.985880 2503 leader_election.cc:398] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 3 election: Vote denied by peer 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203) with higher term. Message: Invalid argument: T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ad91c285f824fc59f71f0e705d419a7 for earlier term 3. Current term is 4.
I20260925 18:11:42.986249 2503 leader_election.cc:401] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260925 18:11:42.986876 2837 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Advancing to term 4
W20260925 18:11:42.982988 2536 rpcz_store.cc:269] 0925 18:11:42.694005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:11:42.695805 (+ 1800us) service_pool.cc:227] Handling call
0925 18:11:42.982441 (+286636us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b5400410354. stack: 00007f5cf09d6033 00007f5cf09d5f30 00007f5cf0334534 00000000003b4464 00000000003b43f9 00007f5cf71ee471 00007f5cf726d2ae 00007f5cf9476c27 00007f5cf6dfec04 00007f5cf6dfeb92 00007f5cf6dfeac1 00007f5cf6dfea3c 00007f5cf6dfdcf2 00007f5cf553694c 00007f5cf5535d86 00007f5cf553572f
0925 18:11:42.982536 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":616221440}
I20260925 18:11:42.988826 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:43.040655 2279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 3.
I20260925 18:11:43.041977 2385 leader_election.cc:302] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c2641b1ebada4ee38658e3bf1fa31415, fc858f182cf448a3a94ba08cb41d69c0; no voters: 9d2456837bc6407e9e32d5503b0f4870
I20260925 18:11:43.042201 2844 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:43.043044 2832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 18:11:43.044281 2844 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 election: Requested vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:11:43.045362 2615 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:11:43.048594 2693 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 3.
I20260925 18:11:43.050428 2581 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7
I20260925 18:11:43.051283 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:43.051448 2832 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 3 LEADER]: Becoming Leader. State: Replica: fc858f182cf448a3a94ba08cb41d69c0, State: Running, Role: LEADER
I20260925 18:11:43.052016 2837 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203) with higher term. Message: Invalid argument: T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ad91c285f824fc59f71f0e705d419a7 for earlier term 3. Current term is 4.
I20260925 18:11:43.052914 2832 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:43.053797 2484 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260925 18:11:43.055050 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:11:43.051911 2844 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 18:11:43.055878 2693 raft_consensus.cc:2354] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 for earlier term 3. Current term is 4.
I20260925 18:11:43.056043 2536 raft_consensus.cc:2354] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 for earlier term 3. Current term is 4.
I20260925 18:11:43.059283 2582 leader_election.cc:398] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 election: Vote denied by peer 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227) with higher term. Message: Invalid argument: T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 for earlier term 3. Current term is 4.
I20260925 18:11:43.059589 2582 leader_election.cc:401] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260925 18:11:43.060397 2581 leader_election.cc:398] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 election: Vote denied by peer 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203) with higher term. Message: Invalid argument: T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 for earlier term 3. Current term is 4.
I20260925 18:11:43.060321 2844 raft_consensus.cc:686] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 3 LEADER]: Becoming Leader. State: Replica: 9f27b2d63b304859af2501907270d673, State: Running, Role: LEADER
I20260925 18:11:43.061740 2844 consensus_queue.cc:237] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:43.063788 2847 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc858f182cf448a3a94ba08cb41d69c0. Latest consensus state: current_term: 3 leader_uuid: "fc858f182cf448a3a94ba08cb41d69c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:43.064867 2847 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:43.068089 2849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:11:43.077302 2849 catalog_manager.cc:669] Loaded metadata for table test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d]
I20260925 18:11:43.084735 2849 tablet_loader.cc:96] loaded metadata for tablet ab34f9c8e4e74a4e981efa7270e886da (table test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d])
I20260925 18:11:43.086254 2849 tablet_loader.cc:96] loaded metadata for tablet bd005713f7db4ce8aca0658034f5b7cb (table test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d])
I20260925 18:11:43.087559 2849 tablet_loader.cc:96] loaded metadata for tablet df8c60f75ea94608b6072d8e17e3e356 (table test-table [id=8aee2e6e479b43348ab72a03b4dd3d5d])
I20260925 18:11:43.088985 2849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:11:43.094496 2849 catalog_manager.cc:1259] Loaded cluster ID: c1bce0eedc5e42e197d53912a6b6702a
I20260925 18:11:43.094815 2849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:11:43.100371 2849 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:11:43.105072 2849 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: Loaded TSK: 0
I20260925 18:11:43.106612 2849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:11:43.117686 2470 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 18:11:43.137761 2346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 3 FOLLOWER]: Refusing update from remote peer fc858f182cf448a3a94ba08cb41d69c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20260925 18:11:43.139199 2847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 18:11:43.142400 2845 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:43.144433 2845 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:43.145462 2616 raft_consensus.cc:1226] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Rejecting Update request from peer 6ad91c285f824fc59f71f0e705d419a7 for earlier term 2. Current term is 3. Ops: []
I20260925 18:11:43.148902 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:43.150292 2484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc858f182cf448a3a94ba08cb41d69c0. Latest consensus state: current_term: 3 leader_uuid: "fc858f182cf448a3a94ba08cb41d69c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:43.151057 2484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:43.152900 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:11:43.153918 2837 consensus_queue.cc:1046] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 18:11:43.154838 2837 raft_consensus.cc:3032] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 2 LEADER]: Stepping down as leader of term 2
I20260925 18:11:43.155139 2837 raft_consensus.cc:729] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
I20260925 18:11:43.156352 2582 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:11:43.156153 2837 consensus_queue.cc:260] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:43.157833 2837 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:43.160507 2845 raft_consensus.cc:2726] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260925 18:11:43.163774 2536 raft_consensus.cc:1261] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Refusing update from remote peer 9f27b2d63b304859af2501907270d673: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260925 18:11:43.165645 2845 consensus_queue.cc:1035] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 18:11:43.168102 2832 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fc858f182cf448a3a94ba08cb41d69c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:43.168936 2832 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:43.169826 2279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 3 FOLLOWER]: Refusing update from remote peer fc858f182cf448a3a94ba08cb41d69c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20260925 18:11:43.171717 2832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 18:11:43.175606 2564 heartbeater.cc:503] Master 127.0.205.188:44299 was elected leader, sending a full tablet report...
I20260925 18:11:43.180466 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:43.184229 2658 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:11:43.185125 2838 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 18:11:43.188572 2693 raft_consensus.cc:1261] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Refusing update from remote peer 9f27b2d63b304859af2501907270d673: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260925 18:11:43.191735 2484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fc858f182cf448a3a94ba08cb41d69c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:43.192530 2484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:43.194232 2845 consensus_queue.cc:1035] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260925 18:11:43.209182 2720 heartbeater.cc:503] Master 127.0.205.188:44299 was elected leader, sending a full tablet report...
I20260925 18:11:43.218506 2642 heartbeater.cc:503] Master 127.0.205.188:44299 was elected leader, sending a full tablet report...
I20260925 18:11:43.230311 2402 catalog_manager.cc:5495] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 reported cstate change: term changed from 2 to 3, leader changed from 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131) to 9f27b2d63b304859af2501907270d673 (127.0.205.130). New cstate: current_term: 3 leader_uuid: "9f27b2d63b304859af2501907270d673" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } }
I20260925 18:11:43.239759 2839 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc858f182cf448a3a94ba08cb41d69c0. Latest consensus state: current_term: 3 leader_uuid: "fc858f182cf448a3a94ba08cb41d69c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:43.240360 2839 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:43.240882 2839 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fc858f182cf448a3a94ba08cb41d69c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:43.241380 2839 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:43.247920 2615 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 4.
I20260925 18:11:43.249686 2836 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227) with higher term. Message: Invalid argument: T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 for earlier term 3. Current term is 4.
W20260925 18:11:43.262322 2615 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35088 (request call id 37) took 295 ms (client timeout 209 ms). Trace:
W20260925 18:11:43.262773 2615 rpcz_store.cc:269] 0925 18:11:42.966506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:11:42.966815 (+ 309us) service_pool.cc:227] Handling call
0925 18:11:43.262138 (+295323us) spinlock_profiling.cc:243] Waited 77.8 ms on lock 0x7b54003d0254. stack: 00007f5cf09d6033 00007f5cf09d5f30 00007f5cf0334534 00000000003b4464 00000000003b43f9 00007f5cf71ee471 00007f5cf726d2ae 00007f5cf9476c27 00007f5cf6dfec04 00007f5cf6dfeb92 00007f5cf6dfeac1 00007f5cf6dfea3c 00007f5cf6dfdcf2 00007f5cf553694c 00007f5cf5535d86 00007f5cf553572f
0925 18:11:43.262240 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":178930944}
I20260925 18:11:43.266433 2832 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fc858f182cf448a3a94ba08cb41d69c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:43.267287 2832 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:43.270594 2837 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:43.271103 2837 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:43.273105 2837 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:43.274704 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:43.275360 2695 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 4.
I20260925 18:11:43.276510 2503 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters:
I20260925 18:11:43.277474 2837 raft_consensus.cc:2781] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 18:11:43.277839 2837 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:11:43.278143 2837 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:11:43.291675 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:43.292332 2615 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 4.
I20260925 18:11:43.346876 2838 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:43.347424 2838 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:43.349320 2838 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:43.350997 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:11:43.351811 2615 raft_consensus.cc:2379] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74156d4911f84a01b82ea7651f6f13c6 in current term 3: Already voted for candidate 9f27b2d63b304859af2501907270d673 in this term.
I20260925 18:11:43.351429 2837 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:43.351943 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:43.352576 2536 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 3.
I20260925 18:11:43.353513 2837 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 election: Requested vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:43.354291 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:43.354875 2693 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:11:43.354626 2660 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters: 9f27b2d63b304859af2501907270d673
I20260925 18:11:43.355681 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:43.355957 2838 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 18:11:43.356348 2838 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:11:43.356312 2615 raft_consensus.cc:3037] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:11:43.356875 2838 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 2 FOLLOWER]: Advancing to term 3
I20260925 18:11:43.382359 2844 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:11:43.382943 2844 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:43.385248 2844 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:43.385622 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:11:43.385871 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:11:43.386279 2536 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 3.
I20260925 18:11:43.387141 2582 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:11:43.387774 2844 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 18:11:43.388128 2844 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:11:43.388424 2844 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:11:43.451115 2693 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 5.
I20260925 18:11:43.452526 2838 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:43.454356 2695 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 3.
I20260925 18:11:43.456967 2615 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 5.
I20260925 18:11:43.457206 2503 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters:
I20260925 18:11:43.458774 2837 raft_consensus.cc:2781] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 18:11:43.459115 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:11:43.460688 2837 raft_consensus.cc:686] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 5 LEADER]: Becoming Leader. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
I20260925 18:11:43.461694 2837 consensus_queue.cc:237] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:11:43.466130 2838 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 3 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:43.467159 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:11:43.475119 2401 catalog_manager.cc:5495] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 reported cstate change: term changed from 2 to 5, leader changed from 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131) to 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129). New cstate: current_term: 5 leader_uuid: "6ad91c285f824fc59f71f0e705d419a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: HEALTHY } } }
I20260925 18:11:43.546693 2616 raft_consensus.cc:1261] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Refusing update from remote peer 6ad91c285f824fc59f71f0e705d419a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 18:11:43.547825 2837 consensus_queue.cc:1035] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 18:11:43.553119 2536 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 3.
I20260925 18:11:43.555157 2844 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:43.556340 2615 raft_consensus.cc:2354] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74156d4911f84a01b82ea7651f6f13c6 for earlier term 3. Current term is 4.
I20260925 18:11:43.558295 2660 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters:
I20260925 18:11:43.559326 2838 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 18:11:43.560045 2658 leader_election.cc:398] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 3 election: Vote denied by peer 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505) with higher term. Message: Invalid argument: T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74156d4911f84a01b82ea7651f6f13c6 for earlier term 3. Current term is 4.
I20260925 18:11:43.561362 2695 raft_consensus.cc:1261] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Refusing update from remote peer 6ad91c285f824fc59f71f0e705d419a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260925 18:11:43.562950 2840 consensus_queue.cc:1035] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 18:11:43.568087 2844 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 4 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:43.570477 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:11:43.572280 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:11:43.572924 2536 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:11:43.591302 2838 raft_consensus.cc:686] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 3 LEADER]: Becoming Leader. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:11:43.592270 2838 consensus_queue.cc:237] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:43.603416 2401 catalog_manager.cc:5495] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 reported cstate change: term changed from 2 to 3, leader changed from 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129) to 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131). New cstate: current_term: 3 leader_uuid: "74156d4911f84a01b82ea7651f6f13c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: HEALTHY } } }
I20260925 18:11:43.652664 2536 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 4.
I20260925 18:11:43.655251 2582 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters: 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:11:43.656805 2844 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 18:11:43.659613 2844 raft_consensus.cc:686] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 4 LEADER]: Becoming Leader. State: Replica: 9f27b2d63b304859af2501907270d673, State: Running, Role: LEADER
I20260925 18:11:43.660776 2844 consensus_queue.cc:237] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:43.669937 2401 catalog_manager.cc:5495] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 reported cstate change: term changed from 3 to 4, leader changed from 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131) to 9f27b2d63b304859af2501907270d673 (127.0.205.130). New cstate: current_term: 4 leader_uuid: "9f27b2d63b304859af2501907270d673" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20260925 18:11:43.686223 2536 raft_consensus.cc:1226] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Rejecting Update request from peer 74156d4911f84a01b82ea7651f6f13c6 for earlier term 3. Current term is 4. Ops: []
I20260925 18:11:43.688133 2838 consensus_queue.cc:1046] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 18:11:43.688812 2871 raft_consensus.cc:3032] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 3 LEADER]: Stepping down as leader of term 3
I20260925 18:11:43.689211 2871 raft_consensus.cc:729] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:11:43.689803 2871 consensus_queue.cc:260] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:11:43.690644 2871 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:11:43.743490 2401 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44808:
table { table_id: "8aee2e6e479b43348ab72a03b4dd3d5d" table_name: "test-table" } new_table_name: "NewTableName"
I20260925 18:11:43.745467 2536 raft_consensus.cc:1261] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Refusing update from remote peer 9f27b2d63b304859af2501907270d673: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260925 18:11:43.746815 2844 consensus_queue.cc:1035] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 18:11:43.782805 2693 raft_consensus.cc:1261] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Refusing update from remote peer 9f27b2d63b304859af2501907270d673: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260925 18:11:43.783236 2693 pending_rounds.cc:76] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6: Aborting all ops after (but not including) 3
I20260925 18:11:43.783481 2693 pending_rounds.cc:97] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20260925 18:11:43.783736 2693 raft_consensus.cc:2866] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 18:11:43.785531 2845 consensus_queue.cc:1035] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 18:11:43.788676 2401 catalog_manager.cc:5587] TS 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505) does not have the latest schema for tablet df8c60f75ea94608b6072d8e17e3e356 (table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d]). Expected version 1 got 0
I20260925 18:11:43.810880 2401 catalog_manager.cc:5587] TS 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203) does not have the latest schema for tablet df8c60f75ea94608b6072d8e17e3e356 (table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d]). Expected version 1 got 0
I20260925 18:11:43.814004 2402 catalog_manager.cc:5587] TS 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227) does not have the latest schema for tablet df8c60f75ea94608b6072d8e17e3e356 (table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d]). Expected version 1 got 0
I20260925 18:11:43.818593 822 master.cc:537] Master@127.0.205.188:44299 shutting down...
I20260925 18:11:43.840426 2884 tablet.cc:1722] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260925 18:11:43.843834 2883 tablet.cc:1722] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260925 18:11:43.863566 2885 tablet.cc:1722] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260925 18:11:43.885031 2884 tablet.cc:1722] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260925 18:11:43.919798 2892 tablet.cc:1722] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260925 18:11:43.922848 2883 tablet.cc:1722] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260925 18:11:43.943341 2891 tablet.cc:1722] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260925 18:11:43.947415 2890 tablet.cc:1722] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260925 18:11:43.948256 2889 tablet.cc:1722] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260925 18:11:44.112676 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 3 LEADER]: Raft consensus shutting down.
I20260925 18:11:44.113492 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260925 18:11:44.113831 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: stopping tablet replica
I20260925 18:11:44.248576 2896 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fc858f182cf448a3a94ba08cb41d69c0)
I20260925 18:11:44.249049 2896 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:44.250639 2896 leader_election.cc:288] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:11:44.251513 2346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
W20260925 18:11:44.256237 2255 leader_election.cc:334] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299): Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111)
I20260925 18:11:44.256626 2255 leader_election.cc:302] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2641b1ebada4ee38658e3bf1fa31415; no voters: 9d2456837bc6407e9e32d5503b0f4870, fc858f182cf448a3a94ba08cb41d69c0
I20260925 18:11:44.257340 2896 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 18:11:44.307904 2899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fc858f182cf448a3a94ba08cb41d69c0)
I20260925 18:11:44.308493 2899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:44.310676 2899 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:11:44.312181 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415" is_pre_election: true
I20260925 18:11:44.312860 2279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2456837bc6407e9e32d5503b0f4870 in term 3.
I20260925 18:11:44.315065 2320 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2456837bc6407e9e32d5503b0f4870, c2641b1ebada4ee38658e3bf1fa31415; no voters:
I20260925 18:11:44.315811 2899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 18:11:44.316150 2899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 3 FOLLOWER]: Starting leader election (detected failure of leader fc858f182cf448a3a94ba08cb41d69c0)
I20260925 18:11:44.316483 2899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 3 FOLLOWER]: Advancing to term 4
W20260925 18:11:44.317389 2320 leader_election.cc:334] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299): Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111)
I20260925 18:11:44.380754 2899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:44.382802 2899 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 4 election: Requested vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:11:44.383572 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415"
I20260925 18:11:44.384114 2279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 3 FOLLOWER]: Advancing to term 4
W20260925 18:11:44.387174 2320 leader_election.cc:334] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299): Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111)
I20260925 18:11:44.394764 2279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2456837bc6407e9e32d5503b0f4870 in term 4.
I20260925 18:11:44.395988 2320 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d2456837bc6407e9e32d5503b0f4870, c2641b1ebada4ee38658e3bf1fa31415; no voters: fc858f182cf448a3a94ba08cb41d69c0
I20260925 18:11:44.396813 2899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 18:11:44.397847 2899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 4 LEADER]: Becoming Leader. State: Replica: 9d2456837bc6407e9e32d5503b0f4870, State: Running, Role: LEADER
I20260925 18:11:44.398674 2899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:44.407043 2902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2456837bc6407e9e32d5503b0f4870. Latest consensus state: current_term: 4 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:44.407838 2902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:44.409981 2903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:11:44.414956 2903 catalog_manager.cc:669] Loaded metadata for table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d]
I20260925 18:11:44.419423 2903 tablet_loader.cc:96] loaded metadata for tablet ab34f9c8e4e74a4e981efa7270e886da (table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d])
I20260925 18:11:44.420387 2903 tablet_loader.cc:96] loaded metadata for tablet bd005713f7db4ce8aca0658034f5b7cb (table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d])
I20260925 18:11:44.421345 2903 tablet_loader.cc:96] loaded metadata for tablet df8c60f75ea94608b6072d8e17e3e356 (table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d])
I20260925 18:11:44.422571 2903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:11:44.426666 2903 catalog_manager.cc:1259] Loaded cluster ID: c1bce0eedc5e42e197d53912a6b6702a
I20260925 18:11:44.426869 2903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:11:44.431308 2903 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:11:44.434484 2903 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870: Loaded TSK: 0
I20260925 18:11:44.435539 2903 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 18:11:44.485867 2320 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 -> Peer fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299): Couldn't send request to peer fc858f182cf448a3a94ba08cb41d69c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 18:11:44.523065 2279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Refusing update from remote peer 9d2456837bc6407e9e32d5503b0f4870: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 4 index: 19. (index mismatch)
I20260925 18:11:44.524626 2902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260925 18:11:44.527848 2896 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2456837bc6407e9e32d5503b0f4870. Latest consensus state: current_term: 4 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:44.528402 2896 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:44.532343 2896 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:44.533125 2896 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:44.531438 2899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:44.533756 2899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:44.598520 2721 heartbeater.cc:503] Master 127.0.205.189:37349 was elected leader, sending a full tablet report...
I20260925 18:11:44.611464 2565 heartbeater.cc:503] Master 127.0.205.189:37349 was elected leader, sending a full tablet report...
I20260925 18:11:44.614054 2643 heartbeater.cc:503] Master 127.0.205.189:37349 was elected leader, sending a full tablet report...
I20260925 18:11:44.626658 2335 catalog_manager.cc:6040] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870: NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d] alter complete (version 1)
W20260925 18:11:44.980201 2320 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 -> Peer fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299): Couldn't send request to peer fc858f182cf448a3a94ba08cb41d69c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260925 18:11:45.137959 822 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 2433)
W20260925 18:11:45.484532 2320 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 -> Peer fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299): Couldn't send request to peer fc858f182cf448a3a94ba08cb41d69c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260925 18:11:46.004961 2320 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 -> Peer fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299): Couldn't send request to peer fc858f182cf448a3a94ba08cb41d69c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260925 18:11:46.138409 822 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 2433)
W20260925 18:11:46.492121 2320 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 -> Peer fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299): Couldn't send request to peer fc858f182cf448a3a94ba08cb41d69c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.205.188:44299: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260925 18:11:46.509579 822 master.cc:552] Master@127.0.205.188:44299 shutdown complete.
I20260925 18:11:46.725055 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:11:46.731547 2912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:46.732362 2911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:11:46.733438 2914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:11:46.734017 822 server_base.cc:1034] running on GCE node
I20260925 18:11:46.735613 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:11:46.735823 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:11:46.735984 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359906735973 us; error 0 us; skew 500 ppm
I20260925 18:11:46.736477 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:11:46.738957 822 webserver.cc:460] Webserver started at http://127.0.205.188:46517/ using document root <none> and password file <none>
I20260925 18:11:46.739425 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:11:46.739589 822 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260925 18:11:46.744992 822 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 18:11:46.747879 2919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:11:46.748875 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 18:11:46.749174 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-2-root
uuid: "fc858f182cf448a3a94ba08cb41d69c0"
format_stamp: "Formatted at 2026-09-25 18:11:36 on dist-test-slave-57m3"
I20260925 18:11:46.749620 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:11:46.774364 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:11:46.775820 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:11:46.822640 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.188:44299
I20260925 18:11:46.822755 2970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.188:44299 every 8 connection(s)
I20260925 18:11:46.833997 2972 sys_catalog.cc:263] Verifying existing consensus state
I20260925 18:11:46.838653 2972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: Bootstrap starting.
I20260925 18:11:46.868579 2564 heartbeater.cc:347] Connected to a master server at 127.0.205.188:44299
I20260925 18:11:46.872778 2720 heartbeater.cc:347] Connected to a master server at 127.0.205.188:44299
I20260925 18:11:46.876638 2642 heartbeater.cc:347] Connected to a master server at 127.0.205.188:44299
I20260925 18:11:46.915195 2972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=7 ignored=0} mutations{seen=14 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 18:11:46.916040 2972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: Bootstrap complete.
I20260925 18:11:46.918555 2972 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:46.919211 2972 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc858f182cf448a3a94ba08cb41d69c0, State: Initialized, Role: FOLLOWER
I20260925 18:11:46.919868 2972 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 3.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:11:46.924769 2972 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:11:46.924356 2977 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:46.925168 2946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:11:46.925256 2977 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:46.938176 2564 heartbeater.cc:347] Connected to a master server at 127.0.205.188:44299
I20260925 18:11:46.960228 2977 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2456837bc6407e9e32d5503b0f4870. Latest consensus state: current_term: 4 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:46.960901 2977 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:46.961926 2985 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:46.962493 2985 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:11:46.968895 2992 catalog_manager.cc:1259] Loaded cluster ID: c1bce0eedc5e42e197d53912a6b6702a
I20260925 18:11:46.969136 2992 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: loading cluster ID for follower catalog manager: success
I20260925 18:11:46.975045 2992 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: acquiring CA information for follower catalog manager: success
I20260925 18:11:46.979609 2992 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 18:11:47.026021 2899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:11:47.027468 2899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: LEADER
I20260925 18:11:47.047183 2935 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" instance_seqno: 1790359896799191) as {username='slave'} at 127.0.0.1:46032; Asking this server to re-register.
I20260925 18:11:47.048301 2720 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:47.050994 2935 ts_manager.cc:194] Registered new tserver with Master: 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:11:47.059233 2935 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "9f27b2d63b304859af2501907270d673" instance_seqno: 1790359896634257) as {username='slave'} at 127.0.0.1:46014; Asking this server to re-register.
I20260925 18:11:47.060143 2936 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" instance_seqno: 1790359896459095) as {username='slave'} at 127.0.0.1:46026; Asking this server to re-register.
I20260925 18:11:47.060495 2642 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:47.061259 2564 heartbeater.cc:464] Registering TS with master...
I20260925 18:11:47.064028 2936 ts_manager.cc:194] Registered new tserver with Master: 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:11:47.064384 2935 ts_manager.cc:194] Registered new tserver with Master: 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:12:02.870823 3014 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9f27b2d63b304859af2501907270d673)
I20260925 18:12:02.871749 3014 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:02.876406 3014 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:02.887856 3025 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9d2456837bc6407e9e32d5503b0f4870)
I20260925 18:12:02.888589 3025 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:02.893119 3025 leader_election.cc:288] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349)
I20260925 18:12:02.909101 3029 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:02.909595 3029 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:12:02.913705 3029 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:12:02.920284 3029 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9f27b2d63b304859af2501907270d673)
I20260925 18:12:02.921109 3029 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:02.923648 3029 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:12:02.932183 3029 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9f27b2d63b304859af2501907270d673)
I20260925 18:12:02.932909 3029 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:02.935344 3029 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:12:02.939531 3031 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9d2456837bc6407e9e32d5503b0f4870)
I20260925 18:12:02.940003 3031 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:02.943400 3031 leader_election.cc:288] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
W20260925 18:12:03.130688 2658 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.205.130:43505 timed out after 0.217s (SENT) [suppressed 34 similar messages]
W20260925 18:12:03.212918 2660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.129:46227, user_credentials={real_user=slave}} blocked reactor thread for 81931.7us
I20260925 18:12:03.214282 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:03.214946 2346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:12:03.215481 2695 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 4.
I20260925 18:12:03.216785 2503 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters:
I20260925 18:12:03.217481 3014 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 18:12:03.217819 3014 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9f27b2d63b304859af2501907270d673)
I20260925 18:12:03.218152 3014 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:12:03.228432 3014 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:03.235458 3014 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 election: Requested vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:03.236518 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:12:03.237176 2694 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:12:03.236547 2536 raft_consensus.cc:1226] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 5 FOLLOWER]: Rejecting Update request from peer 9f27b2d63b304859af2501907270d673 for earlier term 4. Current term is 5. Ops: []
I20260925 18:12:03.240280 2346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:12:03.244261 3014 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9f27b2d63b304859af2501907270d673)
I20260925 18:12:03.249185 3014 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:03.253490 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:03.254115 2695 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 3.
I20260925 18:12:03.255174 2503 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters:
I20260925 18:12:03.249974 2694 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 5.
I20260925 18:12:03.256783 2503 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters:
I20260925 18:12:03.265297 3014 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:03.274155 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:03.559405 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:03.707748 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:03.708601 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:12:03.753614 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:04.118006 3037 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260925 18:12:04.274713 2616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35086 (request call id 238) took 661 ms (client timeout 233 ms). Trace:
W20260925 18:12:04.389060 2255 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.188:44299, user_credentials={real_user=slave}} blocked reactor thread for 148204us
I20260925 18:12:04.228379 3037 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9f27b2d63b304859af2501907270d673)
W20260925 18:12:04.281383 2323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.844s user 0.000s sys 0.013s
W20260925 18:12:04.401394 2323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.994s user 0.001s sys 0.013s
W20260925 18:12:04.274408 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08001d5060 after lost signal to thread 825
I20260925 18:12:04.403208 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
W20260925 18:12:04.404131 2617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35086 (request call id 240) took 284 ms (client timeout 231 ms). Trace:
W20260925 18:12:04.404492 2617 rpcz_store.cc:269] 0925 18:12:04.156503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:04.359834 (+203331us) service_pool.cc:227] Handling call
0925 18:12:04.404068 (+ 44234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 18:12:04.401625 3037 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:12:04.389016 2538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
W20260925 18:12:04.409859 2279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52760 (request call id 1) took 1190 ms (client timeout 239 ms). Trace:
W20260925 18:12:04.410277 2279 rpcz_store.cc:269] 0925 18:12:03.219497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:04.409492 (+1189995us) service_pool.cc:214] Skipping call since client already timed out
0925 18:12:04.409617 (+ 125us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 18:12:04.410693 2538 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47306 (request call id 47) took 702 ms (client timeout 261 ms). Trace:
W20260925 18:12:04.411093 2538 rpcz_store.cc:269] 0925 18:12:03.708074 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:03.824569 (+116495us) service_pool.cc:227] Handling call
0925 18:12:04.410602 (+586033us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 18:12:04.411846 2336 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52858 (request call id 233) took 1474 ms. Trace:
W20260925 18:12:04.389639 2616 rpcz_store.cc:269] 0925 18:12:03.565837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:03.591322 (+ 25485us) service_pool.cc:227] Handling call
0925 18:12:04.227361 (+636039us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 18:12:04.486729 2336 rpcz_store.cc:276] 0925 18:12:02.937373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:04.410425 (+1473052us) service_pool.cc:227] Handling call
0925 18:12:04.411677 (+ 1252us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":0}
I20260925 18:12:04.419413 2535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
W20260925 18:12:04.488854 2535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47306 (request call id 46) took 898 ms (client timeout 212 ms). Trace:
W20260925 18:12:04.489385 2535 rpcz_store.cc:269] 0925 18:12:03.590590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:03.709187 (+118597us) service_pool.cc:227] Handling call
0925 18:12:04.488756 (+779569us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 18:12:04.416383 3037 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:04.490857 2536 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74156d4911f84a01b82ea7651f6f13c6 in current term 4: Already voted for candidate 6ad91c285f824fc59f71f0e705d419a7 in this term.
W20260925 18:12:04.441303 2502 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.130:43505, user_credentials={real_user=slave}} blocked reactor thread for 616132us
I20260925 18:12:04.499926 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:12:04.532789 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:04.533737 2616 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 5.
I20260925 18:12:04.593703 3037 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 4 election: Requested vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:04.595446 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:12:04.597337 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:04.597800 2614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:04.598563 2611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:04.599171 2613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
W20260925 18:12:04.599778 2658 leader_election.cc:334] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505): Timed out: RequestConsensusVote RPC to 127.0.205.130:43505 timed out after 0.212s (SENT)
I20260925 18:12:04.600282 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
W20260925 18:12:04.600430 2658 leader_election.cc:334] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505): Timed out: RequestConsensusVote RPC to 127.0.205.130:43505 timed out after 0.217s (SENT)
I20260925 18:12:04.598536 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:04.596957 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:04.604961 3021 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:04.605475 3021 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:12:04.608090 3021 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
W20260925 18:12:04.608575 2658 leader_election.cc:334] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505): Timed out: RequestConsensusVote RPC to 127.0.205.130:43505 timed out after 0.261s (SENT)
W20260925 18:12:04.710655 2255 leader_election.cc:334] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299): Timed out: RequestConsensusVote RPC to 127.0.205.188:44299 timed out after 0.273s (SENT)
I20260925 18:12:04.711150 2255 leader_election.cc:302] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2641b1ebada4ee38658e3bf1fa31415; no voters: 9d2456837bc6407e9e32d5503b0f4870, fc858f182cf448a3a94ba08cb41d69c0
I20260925 18:12:04.731254 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:12:04.733022 2503 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7; no voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673
I20260925 18:12:04.734179 3037 raft_consensus.cc:2726] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260925 18:12:04.938228 3037 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:12:04.938807 3037 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:04.941902 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:04.945194 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:04.946660 2503 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7; no voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673
I20260925 18:12:04.947566 3038 raft_consensus.cc:2726] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 18:12:04.955441 3043 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 18:12:04.956471 2255 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.188:44299, user_credentials={real_user=slave}} blocked reactor thread for 245977us
W20260925 18:12:04.964046 2539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47306 (request call id 45) took 1398 ms (client timeout 217 ms). Trace:
W20260925 18:12:04.964490 2539 rpcz_store.cc:269] 0925 18:12:03.565385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:04.963716 (+1398331us) service_pool.cc:214] Skipping call since client already timed out
0925 18:12:04.963872 (+ 156us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 18:12:04.989384 2535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:04.989547 3021 consensus_queue.cc:1046] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 18:12:04.996524 3037 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:05.011586 3045 raft_consensus.cc:3032] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 18:12:05.012032 3045 raft_consensus.cc:729] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9f27b2d63b304859af2501907270d673, State: Running, Role: LEADER
W20260925 18:12:05.012809 2695 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48910 (request call id 241) took 408 ms (client timeout 230 ms). Trace:
W20260925 18:12:05.013072 2581 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.131:42203, user_credentials={real_user=slave}} blocked reactor thread for 155653us
W20260925 18:12:05.013221 2695 rpcz_store.cc:269] 0925 18:12:04.604002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:05.012623 (+408621us) service_pool.cc:214] Skipping call since client already timed out
0925 18:12:05.012733 (+ 110us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 18:12:05.014346 2660 leader_election.cc:334] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227): Timed out: RequestConsensusVote RPC to 127.0.205.129:46227 timed out after 0.212s (SENT)
I20260925 18:12:05.014873 2660 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:12:05.015143 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:05.016398 2581 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:12:05.017400 3021 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:05.017539 3045 consensus_queue.cc:260] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:05.018743 3045 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Advancing to term 5
W20260925 18:12:05.021728 2660 leader_election.cc:334] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227): Timed out: RequestConsensusVote RPC to 127.0.205.129:46227 timed out after 0.217s (SENT)
I20260925 18:12:05.021763 3046 raft_consensus.cc:2726] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 18:12:05.022320 2660 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
W20260925 18:12:05.023483 2660 leader_election.cc:334] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227): Timed out: RequestConsensusVote RPC to 127.0.205.129:46227 timed out after 0.261s (SENT)
I20260925 18:12:05.023871 2660 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:12:05.024758 3046 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:05.025513 3046 raft_consensus.cc:2726] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260925 18:12:05.031070 2269 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:60470 (request call id 231) took 1466 ms. Trace:
I20260925 18:12:05.031359 2269 rpcz_store.cc:276] 0925 18:12:03.567537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:03.589505 (+ 21968us) service_pool.cc:227] Handling call
0925 18:12:05.028069 (+1438564us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b5400006fa8. stack: 00007f5cf09d6033 00007f5cf09d5f30 00007f5cf0334534 00000000003b4464 00000000003b43f9 00000000003b4291 00000000003ac78c 00007f5cf988ad54 00007f5cf9984c3e 00007f5cf7c90fb4 00007f5cf7c90f42 00007f5cf7c90e71 00007f5cf7c90dec 00007f5cf7c900a2 00007f5cf553694c 00007f5cf5535d86
0925 18:12:05.030914 (+ 2845us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":56425856}
I20260925 18:12:05.032930 2693 raft_consensus.cc:1226] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Rejecting Update request from peer 9f27b2d63b304859af2501907270d673 for earlier term 4. Current term is 5. Ops: []
W20260925 18:12:05.042686 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08001d57c0 after lost signal to thread 2305
W20260925 18:12:05.043637 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08001d5e80 after lost signal to thread 2370
W20260925 18:12:05.044909 2638 debug-util.cc:398] Leaking SignalData structure 0x7b0800158320 after lost signal to thread 2563
W20260925 18:12:05.045168 2638 debug-util.cc:398] Leaking SignalData structure 0x7b0800143040 after lost signal to thread 2564
W20260925 18:12:05.045920 2638 debug-util.cc:398] Leaking SignalData structure 0x7b080015fa60 after lost signal to thread 2641
W20260925 18:12:05.052174 2536 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47306 (request call id 43) took 1493 ms (client timeout 261 ms). Trace:
I20260925 18:12:05.052258 2539 raft_consensus.cc:1226] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Rejecting Update request from peer 9f27b2d63b304859af2501907270d673 for earlier term 3. Current term is 4. Ops: []
W20260925 18:12:05.052685 2536 rpcz_store.cc:269] 0925 18:12:03.559021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:03.559142 (+ 121us) service_pool.cc:227] Handling call
0925 18:12:05.051929 (+1492787us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b5400420254. stack: 00007f5cf09d6033 00007f5cf09d5f30 00007f5cf0334534 00000000003b4464 00000000003b43f9 00007f5cf71ee471 00007f5cf726d2ae 00007f5cf9476c27 00007f5cf6dfec04 00007f5cf6dfeb92 00007f5cf6dfeac1 00007f5cf6dfea3c 00007f5cf6dfdcf2 00007f5cf553694c 00007f5cf5535d86 00007f5cf553572f
0925 18:12:05.052077 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2136864000}
W20260925 18:12:05.056041 2692 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48914 (request call id 408) took 445 ms (client timeout 249 ms). Trace:
W20260925 18:12:05.056469 2692 rpcz_store.cc:269] 0925 18:12:04.610305 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:05.055717 (+445412us) service_pool.cc:214] Skipping call since client already timed out
0925 18:12:05.055837 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 18:12:05.059237 3045 consensus_queue.cc:1046] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260925 18:12:05.056622 2322 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.711s user 0.013s sys 0.011s
I20260925 18:12:05.060199 3021 raft_consensus.cc:3032] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 3 LEADER]: Stepping down as leader of term 3
W20260925 18:12:05.060365 2322 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.715s user 0.013s sys 0.011s
I20260925 18:12:05.060485 3021 raft_consensus.cc:729] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9f27b2d63b304859af2501907270d673, State: Running, Role: LEADER
I20260925 18:12:05.061348 3021 consensus_queue.cc:260] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:05.062635 3021 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:12:03.265797 3014 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 18:12:05.063417 3014 raft_consensus.cc:686] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 5 LEADER]: Becoming Leader. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
W20260925 18:12:05.063696 2923 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 1.840s user 0.055s sys 0.085s
W20260925 18:12:05.064030 2923 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 1.840s user 0.055s sys 0.085s
I20260925 18:12:05.064308 3014 consensus_queue.cc:237] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:05.066084 2946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 5 candidate_status { last_received { term: 4 index: 20 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0" is_pre_election: true
W20260925 18:12:05.072111 2920 leader_election.cc:334] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227): Timed out: RequestConsensusVote RPC to 127.0.205.190:35227 timed out after 0.239s (SENT)
I20260925 18:12:05.072606 2920 leader_election.cc:302] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc858f182cf448a3a94ba08cb41d69c0; no voters: 9d2456837bc6407e9e32d5503b0f4870, c2641b1ebada4ee38658e3bf1fa31415
W20260925 18:12:05.074126 2638 debug-util.cc:398] Leaking SignalData structure 0x7b0800204c60 after lost signal to thread 2719
W20260925 18:12:05.076467 2638 debug-util.cc:398] Leaking SignalData structure 0x7b0800237ec0 after lost signal to thread 2970
W20260925 18:12:05.077430 2537 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47306 (request call id 44) took 1515 ms (client timeout 212 ms). Trace:
W20260925 18:12:05.078102 2537 rpcz_store.cc:269] 0925 18:12:03.562493 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:03.565888 (+ 3395us) service_pool.cc:227] Handling call
0925 18:12:05.077273 (+1511385us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54003f02d4. stack: 00007f5cf09d6033 00007f5cf09d5f30 00007f5cf0334534 00000000003b4464 00000000003b43f9 00007f5cf71ee471 00007f5cf726d2ae 00007f5cf9476c27 00007f5cf6dfec04 00007f5cf6dfeb92 00007f5cf6dfeac1 00007f5cf6dfea3c 00007f5cf6dfdcf2 00007f5cf553694c 00007f5cf5535d86 00007f5cf553572f
0925 18:12:05.077379 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3032091008}
I20260925 18:12:05.085594 3049 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 18:12:05.085179 2336 catalog_manager.cc:5495] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 reported cstate change: term changed from 4 to 5, leader changed from 9f27b2d63b304859af2501907270d673 (127.0.205.130) to 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129). New cstate: current_term: 5 leader_uuid: "6ad91c285f824fc59f71f0e705d419a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20260925 18:12:05.168937 2616 raft_consensus.cc:1261] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Refusing update from remote peer 6ad91c285f824fc59f71f0e705d419a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260925 18:12:05.170073 3014 consensus_queue.cc:1035] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 18:12:05.180765 2692 raft_consensus.cc:1261] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Refusing update from remote peer 6ad91c285f824fc59f71f0e705d419a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260925 18:12:05.182317 3014 consensus_queue.cc:1035] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 18:12:05.228982 3047 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9f27b2d63b304859af2501907270d673)
I20260925 18:12:05.229489 3047 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:05.231514 3047 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:12:05.236882 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:05.237627 2537 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74156d4911f84a01b82ea7651f6f13c6 in current term 4: Already voted for candidate 6ad91c285f824fc59f71f0e705d419a7 in this term.
I20260925 18:12:05.241221 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:05.241968 2616 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 4.
I20260925 18:12:05.244242 2658 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7
I20260925 18:12:05.255268 3047 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260925 18:12:05.259383 3047 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9f27b2d63b304859af2501907270d673)
I20260925 18:12:05.259859 3047 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 3 FOLLOWER]: Advancing to term 4
I20260925 18:12:05.267127 3047 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:05.268949 3047 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:12:05.269767 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:12:05.270408 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:12:05.270567 2537 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 74156d4911f84a01b82ea7651f6f13c6 in current term 4: Already voted for candidate 6ad91c285f824fc59f71f0e705d419a7 in this term.
I20260925 18:12:05.275418 2616 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 4.
I20260925 18:12:05.277287 2658 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7
I20260925 18:12:05.278473 3047 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Leader election won for term 4
I20260925 18:12:05.279001 3047 raft_consensus.cc:686] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 4 LEADER]: Becoming Leader. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:12:05.279974 3047 consensus_queue.cc:237] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:05.290671 2333 catalog_manager.cc:5495] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 reported cstate change: term changed from 3 to 4, leader changed from 9f27b2d63b304859af2501907270d673 (127.0.205.130) to 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131). New cstate: current_term: 4 leader_uuid: "74156d4911f84a01b82ea7651f6f13c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: HEALTHY } } }
I20260925 18:12:05.346180 3038 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:12:05.346635 3038 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:05.349679 3038 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:05.349579 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:05.349579 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:05.350708 2616 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 4.
I20260925 18:12:05.351786 2502 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters: 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:12:05.352475 3038 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 18:12:05.352814 3038 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:12:05.353120 3038 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:12:05.360386 3038 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:05.362859 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:12:05.363653 2616 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:12:05.363560 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:12:05.373241 3038 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 election: Requested vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:05.377983 2616 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ad91c285f824fc59f71f0e705d419a7 in term 5.
I20260925 18:12:05.378708 2615 raft_consensus.cc:1226] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Rejecting Update request from peer 74156d4911f84a01b82ea7651f6f13c6 for earlier term 4. Current term is 5. Ops: []
I20260925 18:12:05.380093 2502 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters: 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:12:05.380396 3047 consensus_queue.cc:1046] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 18:12:05.380856 3014 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 5 FOLLOWER]: Leader election won for term 5
I20260925 18:12:05.381425 3014 raft_consensus.cc:686] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 5 LEADER]: Becoming Leader. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
I20260925 18:12:05.381623 3046 raft_consensus.cc:3032] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 18:12:05.381950 3046 raft_consensus.cc:729] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:12:05.382706 3046 consensus_queue.cc:260] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:05.384317 3046 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:12:05.383086 3014 consensus_queue.cc:237] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:05.407765 2333 catalog_manager.cc:5495] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 reported cstate change: term changed from 4 to 5, leader changed from 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131) to 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129). New cstate: current_term: 5 leader_uuid: "6ad91c285f824fc59f71f0e705d419a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20260925 18:12:05.492640 2615 raft_consensus.cc:1261] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Refusing update from remote peer 6ad91c285f824fc59f71f0e705d419a7: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260925 18:12:05.492640 2692 raft_consensus.cc:1261] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Refusing update from remote peer 6ad91c285f824fc59f71f0e705d419a7: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260925 18:12:05.493324 2692 pending_rounds.cc:76] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6: Aborting all ops after (but not including) 5
I20260925 18:12:05.493556 2692 pending_rounds.cc:97] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260925 18:12:05.493839 2692 raft_consensus.cc:2866] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 18:12:05.494494 3038 consensus_queue.cc:1035] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 18:12:05.495399 3014 consensus_queue.cc:1035] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260925 18:12:08.654093 3073 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:08.656003 3073 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:12:08.659785 3073 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:12:08.662825 3073 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:08.664021 3073 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:08.666823 3073 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:08.668202 3073 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:08.670012 3073 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:08.672111 3073 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:08.799400 3075 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:08.800149 3075 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:08.801877 3070 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:08.802944 3070 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:12:08.803566 3075 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:12:08.804243 3075 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:08.804950 3070 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:12:08.804733 3075 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:08.805533 3072 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9d2456837bc6407e9e32d5503b0f4870)
I20260925 18:12:08.810132 3072 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:08.818795 3075 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:12:08.838629 3072 leader_election.cc:288] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349)
W20260925 18:12:09.034091 2658 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.205.130:43505 timed out after 0.230s (SENT) [suppressed 13 similar messages]
W20260925 18:12:09.171257 2920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.190:35227, user_credentials={real_user=slave}} blocked reactor thread for 48306.6us
I20260925 18:12:09.547240 2346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:12:09.549242 2346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
W20260925 18:12:09.765403 2581 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.134s user 0.011s sys 0.014s
W20260925 18:12:09.765974 2581 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.134s user 0.012s sys 0.014s
I20260925 18:12:09.796317 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.798153 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.799758 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.802057 3079 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9d2456837bc6407e9e32d5503b0f4870)
I20260925 18:12:09.802588 3079 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
W20260925 18:12:09.803514 2257 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.000s sys 0.004s
W20260925 18:12:09.803875 2257 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.560s user 0.000s sys 0.004s
I20260925 18:12:09.804899 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415" is_pre_election: true
I20260925 18:12:09.805508 2281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 4.
I20260925 18:12:09.807093 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415" is_pre_election: true
I20260925 18:12:09.807665 2281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 4.
I20260925 18:12:09.850966 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.852798 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.864022 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.866164 2346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:12:09.866359 3079 leader_election.cc:288] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:12:09.871287 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.871860 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.872277 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.985000 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.985183 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:09.985800 2537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
W20260925 18:12:10.190585 2583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.538s user 0.000s sys 0.027s
W20260925 18:12:10.191061 2583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.538s user 0.000s sys 0.027s
W20260925 18:12:10.191337 2255 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.188:44299, user_credentials={real_user=slave}} blocked reactor thread for 105015us
W20260925 18:12:10.191891 2920 leader_election.cc:334] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227): Timed out: RequestConsensusVote RPC to 127.0.205.190:35227 timed out after 0.284s (SENT)
I20260925 18:12:10.192121 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
W20260925 18:12:10.194152 2920 leader_election.cc:334] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349): Timed out: RequestConsensusVote RPC to 127.0.205.189:37349 timed out after 0.284s (SENT)
I20260925 18:12:10.192482 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:10.195055 2920 leader_election.cc:302] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc858f182cf448a3a94ba08cb41d69c0; no voters: 9d2456837bc6407e9e32d5503b0f4870, c2641b1ebada4ee38658e3bf1fa31415
I20260925 18:12:10.196853 2945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0" is_pre_election: true
I20260925 18:12:10.196561 3086 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 18:12:10.198601 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:10.199736 2255 leader_election.cc:302] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2641b1ebada4ee38658e3bf1fa31415; no voters: 9d2456837bc6407e9e32d5503b0f4870, fc858f182cf448a3a94ba08cb41d69c0
I20260925 18:12:10.200209 2946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0" is_pre_election: true
I20260925 18:12:10.200745 3079 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260925 18:12:10.203511 2658 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.169s user 0.013s sys 0.020s
W20260925 18:12:10.203847 2658 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.169s user 0.013s sys 0.020s
W20260925 18:12:10.204159 2658 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.130:43505, user_credentials={real_user=slave}} blocked reactor thread for 1.17061e+06us
I20260925 18:12:10.206908 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:10.207600 2692 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 5.
I20260925 18:12:10.209084 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:10.210677 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:10.217424 2693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:10.224092 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:10.225085 2581 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:12:10.225831 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:10.226198 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:10.227593 2658 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:12:10.230237 3088 raft_consensus.cc:2726] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:10.225871 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:10.223577 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:10.230501 2658 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:12:10.234589 3087 raft_consensus.cc:2726] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:10.234884 3088 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:10.235249 2658 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:12:10.235930 3088 raft_consensus.cc:2726] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:10.236630 2581 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:12:10.237324 3087 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:10.237707 2581 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:12:10.238495 2935 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46032 (request call id 197) took 1431 ms. Trace:
I20260925 18:12:10.239064 3087 raft_consensus.cc:2726] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:10.238813 2935 rpcz_store.cc:276] 0925 18:12:08.806595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:08.806771 (+ 176us) service_pool.cc:227] Handling call
0925 18:12:10.237663 (+1430892us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b540048f654. stack: 00007f5cf09d6033 00007f5cf09d5f30 00007f5cf0334534 00000000003b4464 00000000003b43f9 00007f5cf71eeaf0 00007f5cf7271f2a 00007f5cf988ad6c 00007f5cf9984c3e 00007f5cf7c90fb4 00007f5cf7c90f42 00007f5cf7c90e71 00007f5cf7c90dec 00007f5cf7c900a2 00007f5cf553694c 00007f5cf5535d86
0925 18:12:10.238325 (+ 662us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50273664}
W20260925 18:12:10.591446 827 debug-util.cc:398] Leaking SignalData structure 0x7b0800028600 after lost signal to thread 2658
W20260925 18:12:10.666079 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0840 after lost signal to thread 825
W20260925 18:12:10.666924 2638 debug-util.cc:398] Leaking SignalData structure 0x7b0800221ac0 after lost signal to thread 2305
W20260925 18:12:10.667583 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08000b79e0 after lost signal to thread 2370
W20260925 18:12:10.668467 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0d60 after lost signal to thread 2563
W20260925 18:12:10.669211 2638 debug-util.cc:398] Leaking SignalData structure 0x7b0800089700 after lost signal to thread 2641
W20260925 18:12:10.669961 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08000f7760 after lost signal to thread 2719
W20260925 18:12:10.670867 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08000fb8c0 after lost signal to thread 2970
I20260925 18:12:11.248677 3090 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:11.249274 3090 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:12:11.251295 3090 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
I20260925 18:12:11.289273 3090 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:11.289887 3090 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:11.291941 3090 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:12:11.295197 3091 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:11.295734 3091 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:11.297767 3091 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:11.298414 3090 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:11.299567 3090 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:11.301661 3090 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:12:11.302685 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:11.303272 2695 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 5.
I20260925 18:12:11.305079 2581 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:12:11.305644 3091 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 18:12:11.305948 3091 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:11.306262 3091 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Advancing to term 6
I20260925 18:12:11.312662 3091 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:11.315062 3091 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:11.316519 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:12:11.317092 2695 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Advancing to term 6
I20260925 18:12:11.322965 2695 raft_consensus.cc:2454] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 6.
I20260925 18:12:11.324079 3091 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:11.324098 3093 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9d2456837bc6407e9e32d5503b0f4870)
I20260925 18:12:11.324265 2581 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:12:11.324810 3091 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:11.324922 3093 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:11.325415 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:11.326157 2617 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 5.
I20260925 18:12:11.326862 3091 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:11.327389 3091 raft_consensus.cc:2781] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 18:12:11.327593 3093 leader_election.cc:288] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349)
I20260925 18:12:11.327494 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415" is_pre_election: true
I20260925 18:12:11.327876 3091 raft_consensus.cc:686] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 6 LEADER]: Becoming Leader. State: Replica: 9f27b2d63b304859af2501907270d673, State: Running, Role: LEADER
I20260925 18:12:11.328446 2281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 4.
I20260925 18:12:11.328672 3091 consensus_queue.cc:237] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:11.329600 2920 leader_election.cc:302] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2641b1ebada4ee38658e3bf1fa31415, fc858f182cf448a3a94ba08cb41d69c0; no voters:
I20260925 18:12:11.327545 2658 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673; no voters:
I20260925 18:12:11.330677 3090 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 18:12:11.330992 3090 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:11.331308 3090 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Advancing to term 6
I20260925 18:12:11.347116 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:11.347834 2615 raft_consensus.cc:2454] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 5.
W20260925 18:12:11.449000 2581 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.131:42203, user_credentials={real_user=slave}} blocked reactor thread for 124867us
I20260925 18:12:11.484850 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
W20260925 18:12:11.485766 2536 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47306 (request call id 58) took 164 ms (client timeout 215 ms). Trace:
W20260925 18:12:11.486123 2536 rpcz_store.cc:269] 0925 18:12:11.321071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:11.484496 (+163425us) service_pool.cc:227] Handling call
0925 18:12:11.485708 (+ 1212us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260925 18:12:11.530823 2537 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47306 (request call id 56) took 210 ms (client timeout 205 ms). Trace:
W20260925 18:12:11.531402 2537 rpcz_store.cc:269] 0925 18:12:11.320526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:11.530608 (+210082us) service_pool.cc:214] Skipping call since client already timed out
0925 18:12:11.530733 (+ 125us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 18:12:11.559034 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
W20260925 18:12:11.560142 2539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47306 (request call id 57) took 239 ms (client timeout 266 ms). Trace:
W20260925 18:12:11.560619 2539 rpcz_store.cc:269] 0925 18:12:11.320793 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:11.558697 (+237904us) service_pool.cc:227] Handling call
0925 18:12:11.560082 (+ 1385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260925 18:12:11.632042 2692 raft_consensus.cc:1226] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Rejecting Update request from peer 6ad91c285f824fc59f71f0e705d419a7 for earlier term 5. Current term is 6. Ops: []
I20260925 18:12:11.659472 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:11.700223 2539 raft_consensus.cc:3032] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 18:12:11.700798 2539 raft_consensus.cc:729] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
I20260925 18:12:11.701776 2539 consensus_queue.cc:260] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:11.702953 2539 raft_consensus.cc:3037] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 5 FOLLOWER]: Advancing to term 6
I20260925 18:12:11.452669 3090 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:11.796190 2693 raft_consensus.cc:1226] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Rejecting Update request from peer 6ad91c285f824fc59f71f0e705d419a7 for earlier term 5. Current term is 6. Ops: []
I20260925 18:12:11.832340 3093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260925 18:12:11.833038 3093 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9d2456837bc6407e9e32d5503b0f4870)
I20260925 18:12:11.833534 3093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:12:11.877225 3090 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
W20260925 18:12:11.895136 2535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47290 (request call id 415) took 364 ms (client timeout 230 ms). Trace:
W20260925 18:12:11.895746 2535 rpcz_store.cc:269] 0925 18:12:11.530572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:11.894922 (+364350us) service_pool.cc:214] Skipping call since client already timed out
0925 18:12:11.895067 (+ 145us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 18:12:11.910329 2660 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.129:46227, user_credentials={real_user=slave}} blocked reactor thread for 53188.4us
W20260925 18:12:11.917178 2920 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.189:37349, user_credentials={real_user=slave}} blocked reactor thread for 55355.9us
I20260925 18:12:11.917965 3099 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9d2456837bc6407e9e32d5503b0f4870)
I20260925 18:12:11.919173 3099 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:11.921308 2535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:12:11.921489 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
W20260925 18:12:11.922857 2616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35088 (request call id 58) took 597 ms (client timeout 215 ms). Trace:
W20260925 18:12:11.923929 2616 rpcz_store.cc:269] 0925 18:12:11.324966 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:11.922642 (+597676us) service_pool.cc:214] Skipping call since client already timed out
0925 18:12:11.922786 (+ 144us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260925 18:12:11.925114 2346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:12:11.926656 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:11.927215 2535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:11.927628 3092 raft_consensus.cc:482] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9f27b2d63b304859af2501907270d673)
W20260925 18:12:11.928064 2582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.205.129:46227, user_credentials={real_user=slave}} blocked reactor thread for 194033us
I20260925 18:12:11.928120 3092 raft_consensus.cc:504] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:11.929843 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:11.933280 3099 leader_election.cc:288] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:12:11.934552 2535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:12:11.934762 2536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:11.937959 3093 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:11.939777 3093 leader_election.cc:288] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 election: Requested vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349)
I20260925 18:12:11.940594 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415"
I20260925 18:12:11.941471 2346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:12:11.944993 2346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870"
I20260925 18:12:11.946398 2920 leader_election.cc:302] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc858f182cf448a3a94ba08cb41d69c0; no voters: 9d2456837bc6407e9e32d5503b0f4870, c2641b1ebada4ee38658e3bf1fa31415
I20260925 18:12:11.947274 3093 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260925 18:12:11.950156 2945 raft_consensus.cc:1226] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 5 FOLLOWER]: Rejecting Update request from peer 9d2456837bc6407e9e32d5503b0f4870 for earlier term 4. Current term is 5. Ops: []
I20260925 18:12:11.951707 3098 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 }, Status: INVALID_TERM, Last received: 4.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260925 18:12:11.952485 3098 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 4 LEADER]: Stepping down as leader of term 4
I20260925 18:12:11.952706 3098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9d2456837bc6407e9e32d5503b0f4870, State: Running, Role: LEADER
I20260925 18:12:11.953497 3098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 4.23, Last appended by leader: 23, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:11.955085 3098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:12:11.956125 2695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
W20260925 18:12:11.956281 2538 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47290 (request call id 416) took 425 ms (client timeout 280 ms). Trace:
I20260925 18:12:11.956789 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
W20260925 18:12:11.958232 2538 rpcz_store.cc:269] 0925 18:12:11.530874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0925 18:12:11.956028 (+425154us) service_pool.cc:214] Skipping call since client already timed out
0925 18:12:11.956163 (+ 135us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260925 18:12:11.960249 3091 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.003s sys 0.007s
W20260925 18:12:11.960577 3091 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.005s sys 0.008s
I20260925 18:12:11.961421 2946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2641b1ebada4ee38658e3bf1fa31415" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0" is_pre_election: true
I20260925 18:12:11.962109 2946 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c2641b1ebada4ee38658e3bf1fa31415 in current term 5: Already voted for candidate fc858f182cf448a3a94ba08cb41d69c0 in this term.
I20260925 18:12:11.963624 2581 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f27b2d63b304859af2501907270d673; no voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:12:11.964123 2613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:12:11.965421 2612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:11.966107 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd005713f7db4ce8aca0658034f5b7cb" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:11.967330 2617 raft_consensus.cc:1226] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 6 LEADER]: Rejecting Update request from peer 6ad91c285f824fc59f71f0e705d419a7 for earlier term 5. Current term is 6. Ops: []
I20260925 18:12:11.968087 3096 raft_consensus.cc:2726] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:11.967372 3092 leader_election.cc:288] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:11.968305 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:11.969349 2658 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:12:11.971207 3105 raft_consensus.cc:2726] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 18:12:11.974413 2658 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:12:11.974979 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 5 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:12:11.975638 2345 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 5.
I20260925 18:12:11.975864 3090 raft_consensus.cc:2726] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:11.975581 2658 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:12:11.978271 2255 leader_election.cc:302] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2641b1ebada4ee38658e3bf1fa31415; no voters: 9d2456837bc6407e9e32d5503b0f4870, fc858f182cf448a3a94ba08cb41d69c0
I20260925 18:12:11.979202 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df8c60f75ea94608b6072d8e17e3e356" candidate_uuid: "6ad91c285f824fc59f71f0e705d419a7" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:11.979511 3099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 4 FOLLOWER]: Advancing to term 5
I20260925 18:12:11.979653 3090 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:11.985970 3092 consensus_queue.cc:1046] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 18:12:11.987124 3092 raft_consensus.cc:3032] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 5 LEADER]: Stepping down as leader of term 5
I20260925 18:12:11.987784 3092 raft_consensus.cc:729] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6ad91c285f824fc59f71f0e705d419a7, State: Running, Role: LEADER
I20260925 18:12:11.989102 3092 consensus_queue.cc:260] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:11.991499 3092 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 5 FOLLOWER]: Advancing to term 6
I20260925 18:12:11.998404 2502 leader_election.cc:302] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7; no voters: 74156d4911f84a01b82ea7651f6f13c6, 9f27b2d63b304859af2501907270d673
I20260925 18:12:12.003185 3092 raft_consensus.cc:2726] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260925 18:12:12.010267 3099 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260925 18:12:12.192689 3091 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:12.194435 3091 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:12.198717 3091 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:12.200115 3093 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:12:12.200273 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6" is_pre_election: true
I20260925 18:12:12.200863 3093 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:12.201870 2694 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 in current term 6: Already voted for candidate 74156d4911f84a01b82ea7651f6f13c6 in this term.
I20260925 18:12:12.202024 3080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:12:12.202783 3080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:12.203594 3093 leader_election.cc:288] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349)
I20260925 18:12:12.206032 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415" is_pre_election: true
I20260925 18:12:12.207024 2281 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 5.
I20260925 18:12:12.206889 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870" is_pre_election: true
I20260925 18:12:12.208233 2345 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc858f182cf448a3a94ba08cb41d69c0 in term 5.
I20260925 18:12:12.208906 2920 leader_election.cc:302] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2641b1ebada4ee38658e3bf1fa31415, fc858f182cf448a3a94ba08cb41d69c0; no voters:
I20260925 18:12:12.209736 2946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 6 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0" is_pre_election: true
I20260925 18:12:12.209926 3093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 18:12:12.210148 3080 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:12:12.210438 2946 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2456837bc6407e9e32d5503b0f4870 in term 5.
I20260925 18:12:12.210678 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 6 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415" is_pre_election: true
I20260925 18:12:12.211402 2279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2456837bc6407e9e32d5503b0f4870 in term 5.
I20260925 18:12:12.211661 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:12.212419 2539 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 6.
I20260925 18:12:12.212853 2320 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d2456837bc6407e9e32d5503b0f4870, c2641b1ebada4ee38658e3bf1fa31415; no voters:
I20260925 18:12:12.215051 3080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 18:12:12.215672 3080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:12:12.216111 3080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 5 FOLLOWER]: Advancing to term 6
I20260925 18:12:12.217875 2582 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters: 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:12:12.219115 3091 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260925 18:12:12.219539 3091 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:12.220022 3091 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Advancing to term 6
I20260925 18:12:12.220315 3093 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:12:12.220790 3093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 5 FOLLOWER]: Advancing to term 6
I20260925 18:12:12.227906 3080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:12.228402 3091 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:12.231653 3091 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131:42203)
I20260925 18:12:12.232357 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:12:12.239476 3093 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:12.240904 2694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "9f27b2d63b304859af2501907270d673" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "74156d4911f84a01b82ea7651f6f13c6"
I20260925 18:12:12.241598 2694 raft_consensus.cc:2379] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f27b2d63b304859af2501907270d673 in current term 6: Already voted for candidate 74156d4911f84a01b82ea7651f6f13c6 in this term.
I20260925 18:12:12.243546 3080 leader_election.cc:288] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 6 election: Requested vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), fc858f182cf448a3a94ba08cb41d69c0 (127.0.205.188:44299)
I20260925 18:12:12.244174 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 6 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415"
I20260925 18:12:12.244725 2279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 5 FOLLOWER]: Advancing to term 6
I20260925 18:12:12.245726 2946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d2456837bc6407e9e32d5503b0f4870" candidate_term: 6 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "fc858f182cf448a3a94ba08cb41d69c0"
I20260925 18:12:12.246456 2946 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d2456837bc6407e9e32d5503b0f4870 in current term 6: Already voted for candidate fc858f182cf448a3a94ba08cb41d69c0 in this term.
I20260925 18:12:12.246690 3093 leader_election.cc:288] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 6 election: Requested vote from peers c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227), 9d2456837bc6407e9e32d5503b0f4870 (127.0.205.189:37349)
I20260925 18:12:12.244517 2539 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f27b2d63b304859af2501907270d673 in term 6.
I20260925 18:12:12.248093 2582 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673; no voters: 74156d4911f84a01b82ea7651f6f13c6
I20260925 18:12:12.248871 2281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "c2641b1ebada4ee38658e3bf1fa31415"
I20260925 18:12:12.249351 2345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc858f182cf448a3a94ba08cb41d69c0" candidate_term: 6 candidate_status { last_received { term: 4 index: 23 } } ignore_live_leader: false dest_uuid: "9d2456837bc6407e9e32d5503b0f4870"
I20260925 18:12:12.250098 2345 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fc858f182cf448a3a94ba08cb41d69c0 in current term 6: Already voted for candidate 9d2456837bc6407e9e32d5503b0f4870 in this term.
I20260925 18:12:12.251199 2920 leader_election.cc:302] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc858f182cf448a3a94ba08cb41d69c0; no voters: 9d2456837bc6407e9e32d5503b0f4870, c2641b1ebada4ee38658e3bf1fa31415
I20260925 18:12:12.252146 2279 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2456837bc6407e9e32d5503b0f4870 in term 6.
I20260925 18:12:12.252362 3093 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260925 18:12:12.252794 3091 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 18:12:12.253154 2320 leader_election.cc:302] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9d2456837bc6407e9e32d5503b0f4870, c2641b1ebada4ee38658e3bf1fa31415; no voters: fc858f182cf448a3a94ba08cb41d69c0
I20260925 18:12:12.253340 3091 raft_consensus.cc:686] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 6 LEADER]: Becoming Leader. State: Replica: 9f27b2d63b304859af2501907270d673, State: Running, Role: LEADER
I20260925 18:12:12.254926 3091 consensus_queue.cc:237] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:12.256160 3090 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:12:12.253993 3080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 6 FOLLOWER]: Leader election won for term 6
I20260925 18:12:12.256605 3090 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:12.257269 3080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 6 LEADER]: Becoming Leader. State: Replica: 9d2456837bc6407e9e32d5503b0f4870, State: Running, Role: LEADER
I20260925 18:12:12.259052 3090 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:12:12.258771 3080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 4.23, Last appended by leader: 23, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } }
I20260925 18:12:12.260653 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673" is_pre_election: true
I20260925 18:12:12.264513 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7" is_pre_election: true
I20260925 18:12:12.265218 2539 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 6.
I20260925 18:12:12.266359 2660 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters: 9f27b2d63b304859af2501907270d673
I20260925 18:12:12.267560 3090 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260925 18:12:12.267935 3090 raft_consensus.cc:482] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:12:12.268316 3090 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Advancing to term 7
I20260925 18:12:12.280486 3090 raft_consensus.cc:504] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:12.283404 2539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6ad91c285f824fc59f71f0e705d419a7"
I20260925 18:12:12.284050 2539 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 6 FOLLOWER]: Advancing to term 7
I20260925 18:12:12.280532 3098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2456837bc6407e9e32d5503b0f4870. Latest consensus state: current_term: 6 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:12:12.284521 3098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:12.291436 2539 raft_consensus.cc:2454] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74156d4911f84a01b82ea7651f6f13c6 in term 7.
I20260925 18:12:12.292333 3090 leader_election.cc:288] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 7 election: Requested vote from peers 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227), 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505)
I20260925 18:12:12.292590 2615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab34f9c8e4e74a4e981efa7270e886da" candidate_uuid: "74156d4911f84a01b82ea7651f6f13c6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "9f27b2d63b304859af2501907270d673"
I20260925 18:12:12.292636 2660 leader_election.cc:302] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad91c285f824fc59f71f0e705d419a7, 74156d4911f84a01b82ea7651f6f13c6; no voters:
I20260925 18:12:12.293588 3090 raft_consensus.cc:2781] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 7 FOLLOWER]: Leader election won for term 7
I20260925 18:12:12.294057 3090 raft_consensus.cc:686] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 7 LEADER]: Becoming Leader. State: Replica: 74156d4911f84a01b82ea7651f6f13c6, State: Running, Role: LEADER
I20260925 18:12:12.295065 3090 consensus_queue.cc:237] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:12.337831 2539 raft_consensus.cc:1226] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 7 FOLLOWER]: Rejecting Update request from peer 9f27b2d63b304859af2501907270d673 for earlier term 6. Current term is 7. Ops: []
I20260925 18:12:12.338871 3091 consensus_queue.cc:1046] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 18:12:12.339658 3094 raft_consensus.cc:3032] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 6 LEADER]: Stepping down as leader of term 6
I20260925 18:12:12.339911 3094 raft_consensus.cc:729] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 9f27b2d63b304859af2501907270d673, State: Running, Role: LEADER
I20260925 18:12:12.340456 3094 consensus_queue.cc:260] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } }
I20260925 18:12:12.341323 3094 raft_consensus.cc:3037] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 6 FOLLOWER]: Advancing to term 7
I20260925 18:12:12.346866 2279 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 6 FOLLOWER]: Refusing update from remote peer 9d2456837bc6407e9e32d5503b0f4870: Log matching property violated. Preceding OpId in replica: term: 4 index: 23. Preceding OpId from leader: term: 6 index: 24. (index mismatch)
I20260925 18:12:12.348230 3098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260925 18:12:12.352941 3099 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2456837bc6407e9e32d5503b0f4870. Latest consensus state: current_term: 6 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:12:12.353596 3099 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:12:12.356679 3080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:12:12.357662 3080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:12.357894 2946 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 6 FOLLOWER]: Refusing update from remote peer 9d2456837bc6407e9e32d5503b0f4870: Log matching property violated. Preceding OpId in replica: term: 4 index: 23. Preceding OpId from leader: term: 6 index: 24. (index mismatch)
I20260925 18:12:12.359586 3116 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260925 18:12:12.360566 3099 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:12:12.361601 3099 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:12:12.367473 3120 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:12:12.367745 3093 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2456837bc6407e9e32d5503b0f4870. Latest consensus state: current_term: 6 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:12:12.368312 3120 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:12:12.368503 3093 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 18:12:12.408146 2539 raft_consensus.cc:1261] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 7 FOLLOWER]: Refusing update from remote peer 74156d4911f84a01b82ea7651f6f13c6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260925 18:12:12.409336 3090 consensus_queue.cc:1035] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 18:12:12.415731 2615 raft_consensus.cc:1261] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 7 FOLLOWER]: Refusing update from remote peer 74156d4911f84a01b82ea7651f6f13c6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260925 18:12:12.416079 2615 pending_rounds.cc:76] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673: Aborting all ops after (but not including) 6
I20260925 18:12:12.416263 2615 pending_rounds.cc:97] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260925 18:12:12.416496 2615 raft_consensus.cc:2866] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260925 18:12:12.418684 3105 consensus_queue.cc:1035] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260925 18:12:12.471026 3080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "9d2456837bc6407e9e32d5503b0f4870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2641b1ebada4ee38658e3bf1fa31415" member_type: VOTER last_known_addr { host: "127.0.205.190" port: 35227 } } peers { permanent_uuid: "9d2456837bc6407e9e32d5503b0f4870" member_type: VOTER last_known_addr { host: "127.0.205.189" port: 37349 } } peers { permanent_uuid: "fc858f182cf448a3a94ba08cb41d69c0" member_type: VOTER last_known_addr { host: "127.0.205.188" port: 44299 } } }
I20260925 18:12:12.471710 3080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [sys.catalog]: This master's current role is: LEADER
W20260925 18:12:12.583675 2638 debug-util.cc:398] Leaking SignalData structure 0x7b080016b440 after lost signal to thread 825
W20260925 18:12:12.584944 2638 debug-util.cc:398] Leaking SignalData structure 0x7b0800236e80 after lost signal to thread 2305
W20260925 18:12:12.585697 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08000ff840 after lost signal to thread 2370
W20260925 18:12:12.586746 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08001e00a0 after lost signal to thread 2563
W20260925 18:12:12.587575 2638 debug-util.cc:398] Leaking SignalData structure 0x7b0800234de0 after lost signal to thread 2641
W20260925 18:12:12.588418 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08001e09e0 after lost signal to thread 2719
W20260925 18:12:12.589275 2638 debug-util.cc:398] Leaking SignalData structure 0x7b08001395a0 after lost signal to thread 2970
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:783: Failure
Expected equality of these values:
live_row_count
Which is: 802
table_info->GetMetrics()->live_row_count->value()
Which is: 1604
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:767: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260925 18:12:17.728768 3112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:785: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 782, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/ts_tablet_manager-itest.cc:895: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20260925 18:12:17.736083 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:12:17.744240 3112 catalog_manager.cc:669] Loaded metadata for table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d]
I20260925 18:12:17.752234 3112 tablet_loader.cc:96] loaded metadata for tablet ab34f9c8e4e74a4e981efa7270e886da (table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d])
I20260925 18:12:17.753948 3112 tablet_loader.cc:96] loaded metadata for tablet bd005713f7db4ce8aca0658034f5b7cb (table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d])
I20260925 18:12:17.755216 3112 tablet_loader.cc:96] loaded metadata for tablet df8c60f75ea94608b6072d8e17e3e356 (table NewTableName [id=8aee2e6e479b43348ab72a03b4dd3d5d])
I20260925 18:12:17.756873 3112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:12:17.761670 3112 catalog_manager.cc:1259] Loaded cluster ID: c1bce0eedc5e42e197d53912a6b6702a
I20260925 18:12:17.762033 3112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:12:17.768080 3112 catalog_manager.cc:1504] Loading token signing keys...
W20260925 18:12:17.770061 2660 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 24 similar messages]
I20260925 18:12:17.776027 3112 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870: Loaded TSK: 0
I20260925 18:12:17.777376 3112 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 18:12:17.783607 2660 consensus_peers.cc:487] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 -> Peer 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227): Couldn't send request to peer 6ad91c285f824fc59f71f0e705d419a7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260925 18:12:17.789505 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:17.790150 822 tablet_replica.cc:330] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7: stopping tablet replica
I20260925 18:12:17.790674 822 raft_consensus.cc:2229] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 7 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:17.791076 822 raft_consensus.cc:2258] T ab34f9c8e4e74a4e981efa7270e886da P 6ad91c285f824fc59f71f0e705d419a7 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:17.793346 822 tablet_replica.cc:330] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7: stopping tablet replica
I20260925 18:12:17.793792 822 raft_consensus.cc:2229] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 5 LEADER]: Raft consensus shutting down.
I20260925 18:12:17.794543 822 raft_consensus.cc:2258] T bd005713f7db4ce8aca0658034f5b7cb P 6ad91c285f824fc59f71f0e705d419a7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:17.796746 822 tablet_replica.cc:330] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7: stopping tablet replica
I20260925 18:12:17.797222 822 raft_consensus.cc:2229] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:17.797607 822 raft_consensus.cc:2258] T df8c60f75ea94608b6072d8e17e3e356 P 6ad91c285f824fc59f71f0e705d419a7 [term 6 FOLLOWER]: Raft consensus is shut down!
W20260925 18:12:17.801841 2582 consensus_peers.cc:487] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 -> Peer 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227): Couldn't send request to peer 6ad91c285f824fc59f71f0e705d419a7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260925 18:12:17.821311 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:12:17.835162 822 tablet_server.cc:178] TabletServer@127.0.205.130:0 shutting down...
W20260925 18:12:17.848819 2658 consensus_peers.cc:487] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 -> Peer 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505): Couldn't send request to peer 9f27b2d63b304859af2501907270d673. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260925 18:12:17.863113 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:17.863782 822 tablet_replica.cc:330] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673: stopping tablet replica
I20260925 18:12:17.864480 822 raft_consensus.cc:2229] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 6 LEADER]: Raft consensus shutting down.
I20260925 18:12:17.865509 822 raft_consensus.cc:2258] T df8c60f75ea94608b6072d8e17e3e356 P 9f27b2d63b304859af2501907270d673 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:17.867833 822 tablet_replica.cc:330] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673: stopping tablet replica
I20260925 18:12:17.868292 822 raft_consensus.cc:2229] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:17.868726 822 raft_consensus.cc:2258] T bd005713f7db4ce8aca0658034f5b7cb P 9f27b2d63b304859af2501907270d673 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:17.871160 822 tablet_replica.cc:330] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673: stopping tablet replica
I20260925 18:12:17.871567 822 raft_consensus.cc:2229] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 7 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:17.871927 822 raft_consensus.cc:2258] T ab34f9c8e4e74a4e981efa7270e886da P 9f27b2d63b304859af2501907270d673 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:17.891511 2721 heartbeater.cc:503] Master 127.0.205.189:37349 was elected leader, sending a full tablet report...
I20260925 18:12:17.900404 2333 catalog_manager.cc:5495] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 reported cstate change: term changed from 5 to 7, leader changed from 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129) to 74156d4911f84a01b82ea7651f6f13c6 (127.0.205.131). New cstate: current_term: 7 leader_uuid: "74156d4911f84a01b82ea7651f6f13c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } health_report { overall_health: HEALTHY } } }
I20260925 18:12:17.902140 2333 catalog_manager.cc:5495] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 reported cstate change: term changed from 5 to 6, leader changed from 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129) to 9f27b2d63b304859af2501907270d673 (127.0.205.130). New cstate: current_term: 6 leader_uuid: "9f27b2d63b304859af2501907270d673" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } }
I20260925 18:12:17.946710 822 tablet_server.cc:194] TabletServer@127.0.205.130:0 shutdown complete.
I20260925 18:12:17.960597 822 tablet_server.cc:178] TabletServer@127.0.205.131:0 shutting down...
I20260925 18:12:17.962818 3131 raft_consensus.cc:482] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6ad91c285f824fc59f71f0e705d419a7)
I20260925 18:12:17.963361 3131 raft_consensus.cc:504] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74156d4911f84a01b82ea7651f6f13c6" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 42203 } } peers { permanent_uuid: "9f27b2d63b304859af2501907270d673" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 43505 } } peers { permanent_uuid: "6ad91c285f824fc59f71f0e705d419a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 46227 } }
I20260925 18:12:17.965534 3131 leader_election.cc:288] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505), 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227)
W20260925 18:12:17.971763 2658 leader_election.cc:334] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9f27b2d63b304859af2501907270d673 (127.0.205.130:43505): Network error: Client connection negotiation failed: client connection to 127.0.205.130:43505: connect: Connection refused (error 111)
W20260925 18:12:17.973150 2660 leader_election.cc:334] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6ad91c285f824fc59f71f0e705d419a7 (127.0.205.129:46227): Network error: Client connection negotiation failed: client connection to 127.0.205.129:46227: connect: Connection refused (error 111)
I20260925 18:12:17.973623 2660 leader_election.cc:302] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74156d4911f84a01b82ea7651f6f13c6; no voters: 6ad91c285f824fc59f71f0e705d419a7, 9f27b2d63b304859af2501907270d673
I20260925 18:12:17.974677 3131 raft_consensus.cc:2726] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260925 18:12:17.991462 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:17.992075 822 tablet_replica.cc:330] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6: stopping tablet replica
I20260925 18:12:17.992583 822 raft_consensus.cc:2229] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:17.993122 822 raft_consensus.cc:2258] T df8c60f75ea94608b6072d8e17e3e356 P 74156d4911f84a01b82ea7651f6f13c6 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:17.995281 822 tablet_replica.cc:330] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6: stopping tablet replica
I20260925 18:12:17.995707 822 raft_consensus.cc:2229] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:17.996130 822 raft_consensus.cc:2258] T bd005713f7db4ce8aca0658034f5b7cb P 74156d4911f84a01b82ea7651f6f13c6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:17.999137 822 tablet_replica.cc:330] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6: stopping tablet replica
I20260925 18:12:17.999706 822 raft_consensus.cc:2229] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 7 LEADER]: Raft consensus shutting down.
I20260925 18:12:18.000411 822 raft_consensus.cc:2258] T ab34f9c8e4e74a4e981efa7270e886da P 74156d4911f84a01b82ea7651f6f13c6 [term 7 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:18.021896 822 tablet_server.cc:194] TabletServer@127.0.205.131:0 shutdown complete.
I20260925 18:12:18.035764 822 master.cc:537] Master@127.0.205.190:35227 shutting down...
W20260925 18:12:18.053200 2320 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 -> Peer c2641b1ebada4ee38658e3bf1fa31415 (127.0.205.190:35227): Couldn't send request to peer c2641b1ebada4ee38658e3bf1fa31415. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 18:12:18.057492 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:18.058017 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:18.058347 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P c2641b1ebada4ee38658e3bf1fa31415: stopping tablet replica
I20260925 18:12:18.078356 822 master.cc:552] Master@127.0.205.190:35227 shutdown complete.
I20260925 18:12:18.093237 822 master.cc:537] Master@127.0.205.189:37349 shutting down...
I20260925 18:12:18.108927 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 6 LEADER]: Raft consensus shutting down.
I20260925 18:12:18.109735 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:18.110153 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9d2456837bc6407e9e32d5503b0f4870: stopping tablet replica
I20260925 18:12:18.128329 822 master.cc:552] Master@127.0.205.189:37349 shutdown complete.
I20260925 18:12:18.144158 822 master.cc:537] Master@127.0.205.188:44299 shutting down...
I20260925 18:12:18.161036 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 6 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:18.161528 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fc858f182cf448a3a94ba08cb41d69c0 [term 6 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:18.161825 822 tablet_replica.cc:330] stopping tablet replica
I20260925 18:12:18.181445 822 master.cc:552] Master@127.0.205.188:44299 shutdown complete.
I20260925 18:12:18.203181 822 test_util.cc:175] -----------------------------------------------
I20260925 18:12:18.203379 822 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1790359881141916-822-0
[ FAILED ] TsTabletManagerITest.TestTableStats (42349 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20260925 18:12:18.209303 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:43747
I20260925 18:12:18.210325 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:18.216176 3152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:18.216516 3151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:18.217465 3154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:18.218695 822 server_base.cc:1034] running on GCE node
I20260925 18:12:18.219446 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:18.219609 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:18.219712 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359938219704 us; error 0 us; skew 500 ppm
I20260925 18:12:18.220132 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:18.222394 822 webserver.cc:460] Webserver started at http://127.0.205.190:35487/ using document root <none> and password file <none>
I20260925 18:12:18.222805 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:18.222950 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:18.223157 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:18.224236 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "4d359ed15d9e4245838238127f9d23ec"
format_stamp: "Formatted at 2026-09-25 18:12:18 on dist-test-slave-57m3"
I20260925 18:12:18.228389 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 18:12:18.231616 3159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:18.232326 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:12:18.232565 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "4d359ed15d9e4245838238127f9d23ec"
format_stamp: "Formatted at 2026-09-25 18:12:18 on dist-test-slave-57m3"
I20260925 18:12:18.232807 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:18.243973 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:18.245134 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:18.279634 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:43747
I20260925 18:12:18.279726 3210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:43747 every 8 connection(s)
I20260925 18:12:18.283161 3211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:18.292567 3211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec: Bootstrap starting.
I20260925 18:12:18.296784 3211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:18.300697 3211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec: No bootstrap required, opened a new log
I20260925 18:12:18.302625 3211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d359ed15d9e4245838238127f9d23ec" member_type: VOTER }
I20260925 18:12:18.302991 3211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:18.303200 3211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d359ed15d9e4245838238127f9d23ec, State: Initialized, Role: FOLLOWER
I20260925 18:12:18.303714 3211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d359ed15d9e4245838238127f9d23ec" member_type: VOTER }
I20260925 18:12:18.304124 3211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:18.304334 3211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:18.304580 3211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:18.308728 3211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d359ed15d9e4245838238127f9d23ec" member_type: VOTER }
I20260925 18:12:18.309203 3211 leader_election.cc:302] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d359ed15d9e4245838238127f9d23ec; no voters:
I20260925 18:12:18.310184 3211 leader_election.cc:288] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:18.310537 3214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:18.311700 3214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 1 LEADER]: Becoming Leader. State: Replica: 4d359ed15d9e4245838238127f9d23ec, State: Running, Role: LEADER
I20260925 18:12:18.312309 3214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d359ed15d9e4245838238127f9d23ec" member_type: VOTER }
I20260925 18:12:18.312861 3211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:12:18.314949 3215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d359ed15d9e4245838238127f9d23ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d359ed15d9e4245838238127f9d23ec" member_type: VOTER } }
I20260925 18:12:18.315026 3216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d359ed15d9e4245838238127f9d23ec. Latest consensus state: current_term: 1 leader_uuid: "4d359ed15d9e4245838238127f9d23ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d359ed15d9e4245838238127f9d23ec" member_type: VOTER } }
I20260925 18:12:18.315696 3215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:18.315753 3216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:18.318066 3220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:12:18.323112 3220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:12:18.326608 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:12:18.331529 3220 catalog_manager.cc:1347] Generated new cluster ID: 30b46ea32c554692a15ba8c2d8fa04f1
I20260925 18:12:18.331807 3220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:12:18.363099 3220 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:12:18.364733 3220 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:12:18.376341 3220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec: Generated new TSK 0
I20260925 18:12:18.377131 3220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:12:18.393297 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:18.399302 3232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:18.400055 3233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:18.401700 3235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:18.402248 822 server_base.cc:1034] running on GCE node
I20260925 18:12:18.403172 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:18.403344 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:18.403458 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359938403450 us; error 0 us; skew 500 ppm
I20260925 18:12:18.403910 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:18.406181 822 webserver.cc:460] Webserver started at http://127.0.205.129:41893/ using document root <none> and password file <none>
I20260925 18:12:18.406625 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:18.406806 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:18.407014 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:18.408070 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "bb6f731371ac4b1d846eab3233001d55"
format_stamp: "Formatted at 2026-09-25 18:12:18 on dist-test-slave-57m3"
I20260925 18:12:18.412304 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 18:12:18.415839 3240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:18.416498 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:12:18.416757 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "bb6f731371ac4b1d846eab3233001d55"
format_stamp: "Formatted at 2026-09-25 18:12:18 on dist-test-slave-57m3"
I20260925 18:12:18.417057 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:18.428030 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:18.429136 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:18.430465 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:12:18.432559 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:18.432724 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:18.432997 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:18.433151 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:18.469883 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:45985
I20260925 18:12:18.469987 3302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:45985 every 8 connection(s)
I20260925 18:12:18.474334 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:18.482106 3308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:18.483160 3307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:18.483647 3310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:18.484139 822 server_base.cc:1034] running on GCE node
I20260925 18:12:18.485056 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:18.485258 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:18.485440 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359938485424 us; error 0 us; skew 500 ppm
I20260925 18:12:18.485992 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:18.488584 3303 heartbeater.cc:347] Connected to a master server at 127.0.205.190:43747
I20260925 18:12:18.488953 3303 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:18.489208 822 webserver.cc:460] Webserver started at http://127.0.205.130:43255/ using document root <none> and password file <none>
I20260925 18:12:18.489682 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:18.489627 3303 heartbeater.cc:511] Master 127.0.205.190:43747 requested a full tablet report, sending...
I20260925 18:12:18.489945 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:18.490209 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:18.491384 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-1-root/instance:
uuid: "3bc17052934f420a95a561ff6cd625a6"
format_stamp: "Formatted at 2026-09-25 18:12:18 on dist-test-slave-57m3"
I20260925 18:12:18.491690 3176 ts_manager.cc:194] Registered new tserver with Master: bb6f731371ac4b1d846eab3233001d55 (127.0.205.129:45985)
I20260925 18:12:18.494267 3176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58370
I20260925 18:12:18.497267 822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 18:12:18.500425 3315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:18.501225 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:12:18.501489 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-1-root
uuid: "3bc17052934f420a95a561ff6cd625a6"
format_stamp: "Formatted at 2026-09-25 18:12:18 on dist-test-slave-57m3"
I20260925 18:12:18.501741 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:18.554332 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:18.555449 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:18.556882 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:12:18.558956 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:18.559139 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:18.559342 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:18.559476 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:18.598435 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.130:44779
I20260925 18:12:18.598541 3377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.130:44779 every 8 connection(s)
I20260925 18:12:18.602922 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:18.609614 3381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:18.610309 3382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:18.613539 822 server_base.cc:1034] running on GCE node
W20260925 18:12:18.615139 3384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:18.616190 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:18.616444 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:18.616668 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359938616650 us; error 0 us; skew 500 ppm
I20260925 18:12:18.616921 3378 heartbeater.cc:347] Connected to a master server at 127.0.205.190:43747
I20260925 18:12:18.617244 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:18.617314 3378 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:18.618084 3378 heartbeater.cc:511] Master 127.0.205.190:43747 requested a full tablet report, sending...
I20260925 18:12:18.619865 822 webserver.cc:460] Webserver started at http://127.0.205.131:39137/ using document root <none> and password file <none>
I20260925 18:12:18.620188 3176 ts_manager.cc:194] Registered new tserver with Master: 3bc17052934f420a95a561ff6cd625a6 (127.0.205.130:44779)
I20260925 18:12:18.620496 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:18.620738 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:18.621050 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:18.622313 3176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58386
I20260925 18:12:18.622488 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-2-root/instance:
uuid: "b7cb278ed1b24777af2183c877f5f913"
format_stamp: "Formatted at 2026-09-25 18:12:18 on dist-test-slave-57m3"
I20260925 18:12:18.627015 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 18:12:18.630167 3389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:18.630859 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:12:18.631101 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-2-root
uuid: "b7cb278ed1b24777af2183c877f5f913"
format_stamp: "Formatted at 2026-09-25 18:12:18 on dist-test-slave-57m3"
I20260925 18:12:18.631326 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:18.646185 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:18.647243 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:18.648643 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:12:18.650775 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:18.650946 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:18.651171 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:18.651300 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:18.688113 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.131:33841
I20260925 18:12:18.688241 3451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.131:33841 every 8 connection(s)
I20260925 18:12:18.702590 3452 heartbeater.cc:347] Connected to a master server at 127.0.205.190:43747
I20260925 18:12:18.703027 3452 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:18.703698 3452 heartbeater.cc:511] Master 127.0.205.190:43747 requested a full tablet report, sending...
I20260925 18:12:18.705440 3176 ts_manager.cc:194] Registered new tserver with Master: b7cb278ed1b24777af2183c877f5f913 (127.0.205.131:33841)
I20260925 18:12:18.705574 822 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014679162s
I20260925 18:12:18.707010 3176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58390
I20260925 18:12:18.741799 3176 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20260925 18:12:18.743456 3176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 18:12:18.783386 3268 tablet_service.cc:1467] Processing CreateTablet for tablet 5933e48e49c64775b9abe69d4b724999 (DEFAULT_TABLE table=test-table [id=8b0e740a202243dd8c6e9bdc03b25779]), partition=
I20260925 18:12:18.784452 3268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5933e48e49c64775b9abe69d4b724999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:18.794409 3343 tablet_service.cc:1467] Processing CreateTablet for tablet 5933e48e49c64775b9abe69d4b724999 (DEFAULT_TABLE table=test-table [id=8b0e740a202243dd8c6e9bdc03b25779]), partition=
I20260925 18:12:18.795274 3343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5933e48e49c64775b9abe69d4b724999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:18.798938 3468 tablet_bootstrap.cc:492] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55: Bootstrap starting.
I20260925 18:12:18.802975 3417 tablet_service.cc:1467] Processing CreateTablet for tablet 5933e48e49c64775b9abe69d4b724999 (DEFAULT_TABLE table=test-table [id=8b0e740a202243dd8c6e9bdc03b25779]), partition=
I20260925 18:12:18.804055 3417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5933e48e49c64775b9abe69d4b724999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:18.804571 3468 tablet_bootstrap.cc:654] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:18.810180 3470 tablet_bootstrap.cc:492] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Bootstrap starting.
I20260925 18:12:18.810351 3468 tablet_bootstrap.cc:492] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55: No bootstrap required, opened a new log
I20260925 18:12:18.810813 3468 ts_tablet_manager.cc:1397] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260925 18:12:18.813537 3468 raft_consensus.cc:348] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.814291 3468 raft_consensus.cc:374] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:18.814570 3468 raft_consensus.cc:729] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6f731371ac4b1d846eab3233001d55, State: Initialized, Role: FOLLOWER
I20260925 18:12:18.815316 3468 consensus_queue.cc:260] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.816861 3470 tablet_bootstrap.cc:654] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:18.818289 3471 tablet_bootstrap.cc:492] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Bootstrap starting.
I20260925 18:12:18.822690 3470 tablet_bootstrap.cc:492] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: No bootstrap required, opened a new log
I20260925 18:12:18.823208 3470 ts_tablet_manager.cc:1397] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260925 18:12:18.823348 3471 tablet_bootstrap.cc:654] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:18.825359 3468 ts_tablet_manager.cc:1428] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
W20260925 18:12:18.825688 3468 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260925 18:12:18.826261 3303 heartbeater.cc:503] Master 127.0.205.190:43747 was elected leader, sending a full tablet report...
I20260925 18:12:18.825964 3470 raft_consensus.cc:348] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.826689 3470 raft_consensus.cc:374] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:18.827023 3470 raft_consensus.cc:729] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc17052934f420a95a561ff6cd625a6, State: Initialized, Role: FOLLOWER
I20260925 18:12:18.827816 3470 consensus_queue.cc:260] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.829839 3378 heartbeater.cc:503] Master 127.0.205.190:43747 was elected leader, sending a full tablet report...
I20260925 18:12:18.830324 3470 ts_tablet_manager.cc:1428] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260925 18:12:18.830605 3470 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260925 18:12:18.834057 3471 tablet_bootstrap.cc:492] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: No bootstrap required, opened a new log
I20260925 18:12:18.834501 3471 ts_tablet_manager.cc:1397] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.008s
I20260925 18:12:18.836329 3471 raft_consensus.cc:348] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.837020 3471 raft_consensus.cc:374] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:18.837234 3471 raft_consensus.cc:729] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cb278ed1b24777af2183c877f5f913, State: Initialized, Role: FOLLOWER
I20260925 18:12:18.837749 3471 consensus_queue.cc:260] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.839744 3452 heartbeater.cc:503] Master 127.0.205.190:43747 was elected leader, sending a full tablet report...
I20260925 18:12:18.839905 3471 ts_tablet_manager.cc:1428] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260925 18:12:18.840185 3471 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
W20260925 18:12:18.851990 3379 tablet.cc:2369] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 18:12:18.918226 3475 raft_consensus.cc:482] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:12:18.918816 3472 raft_consensus.cc:482] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:12:18.918679 3475 raft_consensus.cc:504] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.919231 3472 raft_consensus.cc:504] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.920607 3475 leader_election.cc:288] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb6f731371ac4b1d846eab3233001d55 (127.0.205.129:45985), b7cb278ed1b24777af2183c877f5f913 (127.0.205.131:33841)
I20260925 18:12:18.921108 3472 leader_election.cc:288] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bc17052934f420a95a561ff6cd625a6 (127.0.205.130:44779), b7cb278ed1b24777af2183c877f5f913 (127.0.205.131:33841)
I20260925 18:12:18.943272 3427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5933e48e49c64775b9abe69d4b724999" candidate_uuid: "3bc17052934f420a95a561ff6cd625a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cb278ed1b24777af2183c877f5f913" is_pre_election: true
I20260925 18:12:18.943326 3353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5933e48e49c64775b9abe69d4b724999" candidate_uuid: "bb6f731371ac4b1d846eab3233001d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc17052934f420a95a561ff6cd625a6" is_pre_election: true
I20260925 18:12:18.944056 3427 raft_consensus.cc:2454] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bc17052934f420a95a561ff6cd625a6 in term 0.
I20260925 18:12:18.944136 3353 raft_consensus.cc:2454] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6f731371ac4b1d846eab3233001d55 in term 0.
I20260925 18:12:18.945259 3318 leader_election.cc:302] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc17052934f420a95a561ff6cd625a6, b7cb278ed1b24777af2183c877f5f913; no voters:
I20260925 18:12:18.945638 3278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5933e48e49c64775b9abe69d4b724999" candidate_uuid: "3bc17052934f420a95a561ff6cd625a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6f731371ac4b1d846eab3233001d55" is_pre_election: true
I20260925 18:12:18.946100 3475 raft_consensus.cc:2781] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 18:12:18.946273 3278 raft_consensus.cc:2454] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bc17052934f420a95a561ff6cd625a6 in term 0.
I20260925 18:12:18.946547 3475 raft_consensus.cc:482] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:12:18.946919 3242 leader_election.cc:302] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bc17052934f420a95a561ff6cd625a6, bb6f731371ac4b1d846eab3233001d55; no voters:
I20260925 18:12:18.946914 3475 raft_consensus.cc:3037] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:18.947703 3472 raft_consensus.cc:2781] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 18:12:18.948168 3472 raft_consensus.cc:482] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:12:18.948467 3472 raft_consensus.cc:3037] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:18.948596 3427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5933e48e49c64775b9abe69d4b724999" candidate_uuid: "bb6f731371ac4b1d846eab3233001d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cb278ed1b24777af2183c877f5f913" is_pre_election: true
I20260925 18:12:18.949280 3427 raft_consensus.cc:2454] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb6f731371ac4b1d846eab3233001d55 in term 0.
I20260925 18:12:18.953094 3475 raft_consensus.cc:504] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.953253 3472 raft_consensus.cc:504] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.954587 3472 leader_election.cc:288] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [CANDIDATE]: Term 1 election: Requested vote from peers 3bc17052934f420a95a561ff6cd625a6 (127.0.205.130:44779), b7cb278ed1b24777af2183c877f5f913 (127.0.205.131:33841)
I20260925 18:12:18.955250 3475 leader_election.cc:288] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [CANDIDATE]: Term 1 election: Requested vote from peers bb6f731371ac4b1d846eab3233001d55 (127.0.205.129:45985), b7cb278ed1b24777af2183c877f5f913 (127.0.205.131:33841)
I20260925 18:12:18.956041 3278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5933e48e49c64775b9abe69d4b724999" candidate_uuid: "3bc17052934f420a95a561ff6cd625a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6f731371ac4b1d846eab3233001d55"
I20260925 18:12:18.955413 3353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5933e48e49c64775b9abe69d4b724999" candidate_uuid: "bb6f731371ac4b1d846eab3233001d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bc17052934f420a95a561ff6cd625a6"
I20260925 18:12:18.956367 3426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5933e48e49c64775b9abe69d4b724999" candidate_uuid: "3bc17052934f420a95a561ff6cd625a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cb278ed1b24777af2183c877f5f913"
I20260925 18:12:18.956924 3353 raft_consensus.cc:2379] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bb6f731371ac4b1d846eab3233001d55 in current term 1: Already voted for candidate 3bc17052934f420a95a561ff6cd625a6 in this term.
I20260925 18:12:18.956826 3427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5933e48e49c64775b9abe69d4b724999" candidate_uuid: "bb6f731371ac4b1d846eab3233001d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7cb278ed1b24777af2183c877f5f913"
I20260925 18:12:18.957062 3426 raft_consensus.cc:3037] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:18.958326 3243 leader_election.cc:302] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb6f731371ac4b1d846eab3233001d55; no voters: 3bc17052934f420a95a561ff6cd625a6, b7cb278ed1b24777af2183c877f5f913
I20260925 18:12:18.958459 3278 raft_consensus.cc:2379] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3bc17052934f420a95a561ff6cd625a6 in current term 1: Already voted for candidate bb6f731371ac4b1d846eab3233001d55 in this term.
I20260925 18:12:18.958983 3472 raft_consensus.cc:2726] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 18:12:18.962203 3426 raft_consensus.cc:2454] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bc17052934f420a95a561ff6cd625a6 in term 1.
I20260925 18:12:18.963035 3318 leader_election.cc:302] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3bc17052934f420a95a561ff6cd625a6, b7cb278ed1b24777af2183c877f5f913; no voters: bb6f731371ac4b1d846eab3233001d55
I20260925 18:12:18.963626 3475 raft_consensus.cc:2781] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:18.964592 3475 raft_consensus.cc:686] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 1 LEADER]: Becoming Leader. State: Replica: 3bc17052934f420a95a561ff6cd625a6, State: Running, Role: LEADER
I20260925 18:12:18.965358 3475 consensus_queue.cc:237] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:18.973289 3176 catalog_manager.cc:5495] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bc17052934f420a95a561ff6cd625a6 (127.0.205.130). New cstate: current_term: 1 leader_uuid: "3bc17052934f420a95a561ff6cd625a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } health_report { overall_health: UNKNOWN } } }
I20260925 18:12:19.024169 3417 tablet_service.cc:1514] Processing DeleteTablet for tablet 5933e48e49c64775b9abe69d4b724999 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:46566
I20260925 18:12:19.025599 3483 tablet_replica.cc:330] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: stopping tablet replica
I20260925 18:12:19.026194 3483 raft_consensus.cc:2229] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:19.026592 3483 raft_consensus.cc:2258] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:19.028434 3483 ts_tablet_manager.cc:1905] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 18:12:19.038213 3483 ts_tablet_manager.cc:1918] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20260925 18:12:19.038579 3483 log.cc:1196] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Deleting WAL directory at /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-2-root/wals/5933e48e49c64775b9abe69d4b724999
I20260925 18:12:19.042342 3484 ts_tablet_manager.cc:927] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Initiating tablet copy from peer 3bc17052934f420a95a561ff6cd625a6 (127.0.205.130:44779)
I20260925 18:12:19.043974 3484 tablet_copy_client.cc:316] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: tablet copy: Beginning tablet copy session from remote peer at address 127.0.205.130:44779
I20260925 18:12:19.054057 3363 tablet_copy_service.cc:140] P 3bc17052934f420a95a561ff6cd625a6: Received BeginTabletCopySession request for tablet 5933e48e49c64775b9abe69d4b724999 from peer b7cb278ed1b24777af2183c877f5f913 ({username='slave'} at 127.0.0.1:56634)
I20260925 18:12:19.054535 3363 tablet_copy_service.cc:161] P 3bc17052934f420a95a561ff6cd625a6: Beginning new tablet copy session on tablet 5933e48e49c64775b9abe69d4b724999 from peer b7cb278ed1b24777af2183c877f5f913 at {username='slave'} at 127.0.0.1:56634: session id = b7cb278ed1b24777af2183c877f5f913-5933e48e49c64775b9abe69d4b724999
I20260925 18:12:19.059588 3363 tablet_copy_source_session.cc:205] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 18:12:19.062081 3484 ts_tablet_manager.cc:1905] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260925 18:12:19.071425 3484 ts_tablet_manager.cc:1918] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20260925 18:12:19.072039 3484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5933e48e49c64775b9abe69d4b724999. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:19.076872 3484 tablet_copy_client.cc:791] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: tablet copy: Starting download of 0 data blocks...
I20260925 18:12:19.077327 3484 tablet_copy_client.cc:655] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: tablet copy: Starting download of 1 WAL segments...
I20260925 18:12:19.078856 3176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58402:
table { table_id: "8b0e740a202243dd8c6e9bdc03b25779" table_name: "test-table" }
I20260925 18:12:19.079528 3176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58402:
table { table_id: "8b0e740a202243dd8c6e9bdc03b25779" table_name: "test-table" }
I20260925 18:12:19.080986 3484 tablet_copy_client.cc:531] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 18:12:19.087405 3484 tablet_bootstrap.cc:492] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Bootstrap starting.
I20260925 18:12:19.088564 3176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec: Sending DeleteTablet for 3 replicas of tablet 5933e48e49c64775b9abe69d4b724999
I20260925 18:12:19.090842 3268 tablet_service.cc:1514] Processing DeleteTablet for tablet 5933e48e49c64775b9abe69d4b724999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 18:12:19 UTC) from {username='slave'} at 127.0.0.1:60606
I20260925 18:12:19.090864 3343 tablet_service.cc:1514] Processing DeleteTablet for tablet 5933e48e49c64775b9abe69d4b724999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 18:12:19 UTC) from {username='slave'} at 127.0.0.1:56596
I20260925 18:12:19.090960 3417 tablet_service.cc:1514] Processing DeleteTablet for tablet 5933e48e49c64775b9abe69d4b724999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 18:12:19 UTC) from {username='slave'} at 127.0.0.1:46536
I20260925 18:12:19.093281 3486 tablet_replica.cc:330] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55: stopping tablet replica
W20260925 18:12:19.094221 3162 catalog_manager.cc:4821] TS b7cb278ed1b24777af2183c877f5f913 (127.0.205.131:33841): delete failed for tablet 5933e48e49c64775b9abe69d4b724999 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5933e48e49c64775b9abe69d4b724999 already in progress: copying tablet
I20260925 18:12:19.100208 3486 raft_consensus.cc:2229] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:19.100773 3486 raft_consensus.cc:2258] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:19.101141 3487 tablet_replica.cc:330] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: stopping tablet replica
I20260925 18:12:19.101902 3487 raft_consensus.cc:2229] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:19.102699 3487 pending_rounds.cc:62] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Trying to abort 1 pending ops.
I20260925 18:12:19.102928 3487 pending_rounds.cc:68] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333314310009643008 op_type: NO_OP noop_request { }
I20260925 18:12:19.103384 3487 raft_consensus.cc:2866] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260925 18:12:19.103708 3487 raft_consensus.cc:2258] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:19.104674 3486 ts_tablet_manager.cc:1905] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 18:12:19.105244 3484 tablet_bootstrap.cc:492] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260925 18:12:19.106175 3484 tablet_bootstrap.cc:492] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Bootstrap complete.
I20260925 18:12:19.106843 3484 ts_tablet_manager.cc:1397] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.004s
I20260925 18:12:19.106819 3487 ts_tablet_manager.cc:1905] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 18:12:19.109445 3484 raft_consensus.cc:348] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:19.110517 3484 raft_consensus.cc:729] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7cb278ed1b24777af2183c877f5f913, State: Initialized, Role: FOLLOWER
I20260925 18:12:19.111419 3484 consensus_queue.cc:260] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6f731371ac4b1d846eab3233001d55" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45985 } } peers { permanent_uuid: "3bc17052934f420a95a561ff6cd625a6" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44779 } } peers { permanent_uuid: "b7cb278ed1b24777af2183c877f5f913" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 33841 } }
I20260925 18:12:19.113597 3484 ts_tablet_manager.cc:1428] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260925 18:12:19.113859 3484 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20260925 18:12:19.116719 3486 ts_tablet_manager.cc:1918] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260925 18:12:19.117174 3486 log.cc:1196] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55: Deleting WAL directory at /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-0-root/wals/5933e48e49c64775b9abe69d4b724999
I20260925 18:12:19.117893 3486 ts_tablet_manager.cc:1939] T 5933e48e49c64775b9abe69d4b724999 P bb6f731371ac4b1d846eab3233001d55: Deleting consensus metadata
I20260925 18:12:19.118853 3417 tablet_service.cc:1514] Processing DeleteTablet for tablet 5933e48e49c64775b9abe69d4b724999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 18:12:19 UTC) from {username='slave'} at 127.0.0.1:46536
W20260925 18:12:19.120456 3162 catalog_manager.cc:4821] TS b7cb278ed1b24777af2183c877f5f913 (127.0.205.131:33841): delete failed for tablet 5933e48e49c64775b9abe69d4b724999 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5933e48e49c64775b9abe69d4b724999 already in progress: copying tablet
I20260925 18:12:19.120805 3161 catalog_manager.cc:4841] TS bb6f731371ac4b1d846eab3233001d55 (127.0.205.129:45985): tablet 5933e48e49c64775b9abe69d4b724999 (table test-table [id=8b0e740a202243dd8c6e9bdc03b25779]) successfully deleted
I20260925 18:12:19.122155 3487 ts_tablet_manager.cc:1918] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260925 18:12:19.122556 3487 log.cc:1196] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Deleting WAL directory at /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-1-root/wals/5933e48e49c64775b9abe69d4b724999
I20260925 18:12:19.123195 3487 ts_tablet_manager.cc:1939] T 5933e48e49c64775b9abe69d4b724999 P 3bc17052934f420a95a561ff6cd625a6: Deleting consensus metadata
I20260925 18:12:19.124297 3161 catalog_manager.cc:4841] TS 3bc17052934f420a95a561ff6cd625a6 (127.0.205.130:44779): tablet 5933e48e49c64775b9abe69d4b724999 (table test-table [id=8b0e740a202243dd8c6e9bdc03b25779]) successfully deleted
I20260925 18:12:19.165860 3363 tablet_copy_service.cc:342] P 3bc17052934f420a95a561ff6cd625a6: Request end of tablet copy session b7cb278ed1b24777af2183c877f5f913-5933e48e49c64775b9abe69d4b724999 received from {username='slave'} at 127.0.0.1:56634
I20260925 18:12:19.166229 3363 tablet_copy_service.cc:433] P 3bc17052934f420a95a561ff6cd625a6: ending tablet copy session b7cb278ed1b24777af2183c877f5f913-5933e48e49c64775b9abe69d4b724999 on tablet 5933e48e49c64775b9abe69d4b724999 with peer b7cb278ed1b24777af2183c877f5f913
I20260925 18:12:19.169399 3417 tablet_service.cc:1514] Processing DeleteTablet for tablet 5933e48e49c64775b9abe69d4b724999 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 18:12:19 UTC) from {username='slave'} at 127.0.0.1:46536
I20260925 18:12:19.170153 3483 tablet_replica.cc:330] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: stopping tablet replica
I20260925 18:12:19.170861 3483 raft_consensus.cc:2229] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:19.171280 3483 pending_rounds.cc:62] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Trying to abort 1 pending ops.
I20260925 18:12:19.171533 3483 pending_rounds.cc:68] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333314310009643008 op_type: NO_OP noop_request { }
I20260925 18:12:19.171919 3483 raft_consensus.cc:2866] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260925 18:12:19.172224 3483 raft_consensus.cc:2258] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:19.174093 3483 ts_tablet_manager.cc:1905] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 18:12:19.183218 3483 ts_tablet_manager.cc:1918] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260925 18:12:19.183497 3483 log.cc:1196] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Deleting WAL directory at /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1790359881141916-822-0/minicluster-data/ts-2-root/wals/5933e48e49c64775b9abe69d4b724999
I20260925 18:12:19.184015 3483 ts_tablet_manager.cc:1939] T 5933e48e49c64775b9abe69d4b724999 P b7cb278ed1b24777af2183c877f5f913: Deleting consensus metadata
I20260925 18:12:19.186108 3162 catalog_manager.cc:4841] TS b7cb278ed1b24777af2183c877f5f913 (127.0.205.131:33841): tablet 5933e48e49c64775b9abe69d4b724999 (table test-table [id=8b0e740a202243dd8c6e9bdc03b25779]) successfully deleted
I20260925 18:12:19.233587 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:12:19.249845 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:19.266405 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:12:19.274497 822 tablet_server.cc:178] TabletServer@127.0.205.130:0 shutting down...
I20260925 18:12:19.290035 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:19.308125 822 tablet_server.cc:194] TabletServer@127.0.205.130:0 shutdown complete.
I20260925 18:12:19.316162 822 tablet_server.cc:178] TabletServer@127.0.205.131:0 shutting down...
I20260925 18:12:19.331112 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:19.347991 822 tablet_server.cc:194] TabletServer@127.0.205.131:0 shutdown complete.
I20260925 18:12:19.356722 822 master.cc:537] Master@127.0.205.190:43747 shutting down...
I20260925 18:12:19.369874 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:19.370370 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:19.370669 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4d359ed15d9e4245838238127f9d23ec: stopping tablet replica
I20260925 18:12:19.378312 822 master.cc:552] Master@127.0.205.190:43747 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1191 ms)
[----------] 4 tests from TsTabletManagerITest (48170 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20260925 18:12:19.400749 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:40137
I20260925 18:12:19.401814 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:19.406504 3493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:19.407455 3494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:19.408380 3496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:19.409157 822 server_base.cc:1034] running on GCE node
I20260925 18:12:19.410027 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:19.410189 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:19.410341 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359939410328 us; error 0 us; skew 500 ppm
I20260925 18:12:19.410840 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:19.413133 822 webserver.cc:460] Webserver started at http://127.0.205.190:44483/ using document root <none> and password file <none>
I20260925 18:12:19.413605 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:19.413764 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:19.414009 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:19.415026 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "b24694fc1eff4cc292b5ab58b42b89a9"
format_stamp: "Formatted at 2026-09-25 18:12:19 on dist-test-slave-57m3"
I20260925 18:12:19.419210 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 18:12:19.422406 3501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:19.423178 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 18:12:19.423491 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "b24694fc1eff4cc292b5ab58b42b89a9"
format_stamp: "Formatted at 2026-09-25 18:12:19 on dist-test-slave-57m3"
I20260925 18:12:19.423808 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:19.457624 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:19.458626 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:19.491370 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:40137
I20260925 18:12:19.491442 3552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:40137 every 8 connection(s)
I20260925 18:12:19.494997 3553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:19.504537 3553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9: Bootstrap starting.
I20260925 18:12:19.508477 3553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:19.512217 3553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9: No bootstrap required, opened a new log
I20260925 18:12:19.514078 3553 raft_consensus.cc:348] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24694fc1eff4cc292b5ab58b42b89a9" member_type: VOTER }
I20260925 18:12:19.514443 3553 raft_consensus.cc:374] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:19.514660 3553 raft_consensus.cc:729] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24694fc1eff4cc292b5ab58b42b89a9, State: Initialized, Role: FOLLOWER
I20260925 18:12:19.515199 3553 consensus_queue.cc:260] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24694fc1eff4cc292b5ab58b42b89a9" member_type: VOTER }
I20260925 18:12:19.515612 3553 raft_consensus.cc:388] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:19.515826 3553 raft_consensus.cc:482] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:19.516069 3553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:19.520205 3553 raft_consensus.cc:504] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24694fc1eff4cc292b5ab58b42b89a9" member_type: VOTER }
I20260925 18:12:19.520686 3553 leader_election.cc:302] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b24694fc1eff4cc292b5ab58b42b89a9; no voters:
I20260925 18:12:19.521796 3553 leader_election.cc:288] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:19.522112 3556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:19.523327 3556 raft_consensus.cc:686] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 1 LEADER]: Becoming Leader. State: Replica: b24694fc1eff4cc292b5ab58b42b89a9, State: Running, Role: LEADER
I20260925 18:12:19.524016 3556 consensus_queue.cc:237] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24694fc1eff4cc292b5ab58b42b89a9" member_type: VOTER }
I20260925 18:12:19.524560 3553 sys_catalog.cc:564] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:12:19.527081 3557 sys_catalog.cc:455] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b24694fc1eff4cc292b5ab58b42b89a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24694fc1eff4cc292b5ab58b42b89a9" member_type: VOTER } }
I20260925 18:12:19.527169 3558 sys_catalog.cc:455] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b24694fc1eff4cc292b5ab58b42b89a9. Latest consensus state: current_term: 1 leader_uuid: "b24694fc1eff4cc292b5ab58b42b89a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24694fc1eff4cc292b5ab58b42b89a9" member_type: VOTER } }
I20260925 18:12:19.527872 3558 sys_catalog.cc:458] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:19.527884 3557 sys_catalog.cc:458] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:19.537103 3562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:12:19.541451 3562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:12:19.542462 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:12:19.549427 3562 catalog_manager.cc:1347] Generated new cluster ID: 290f289ca5b04a56a8512d4ff572ec27
I20260925 18:12:19.549664 3562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:12:19.571136 3562 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:12:19.572736 3562 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:12:19.583353 3562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9: Generated new TSK 0
I20260925 18:12:19.583986 3562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:12:19.608903 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:19.614406 3574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:19.615339 3575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:19.616657 3577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:19.617545 822 server_base.cc:1034] running on GCE node
I20260925 18:12:19.618243 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:19.618419 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:19.618561 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359939618547 us; error 0 us; skew 500 ppm
I20260925 18:12:19.619042 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:19.621270 822 webserver.cc:460] Webserver started at http://127.0.205.129:45779/ using document root <none> and password file <none>
I20260925 18:12:19.621711 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:19.621871 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:19.622097 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:19.623090 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "3ff5b300a7d24f83848d93f0502ad32a"
format_stamp: "Formatted at 2026-09-25 18:12:19 on dist-test-slave-57m3"
I20260925 18:12:19.627491 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 18:12:19.630584 3582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:19.631302 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 18:12:19.631618 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "3ff5b300a7d24f83848d93f0502ad32a"
format_stamp: "Formatted at 2026-09-25 18:12:19 on dist-test-slave-57m3"
I20260925 18:12:19.631886 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:19.656409 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:19.657562 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:19.663406 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:19.663700 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:19.663924 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:19.664067 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:19.710321 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:41021
I20260925 18:12:19.710388 3652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:41021 every 8 connection(s)
I20260925 18:12:19.714733 3653 heartbeater.cc:347] Connected to a master server at 127.0.205.190:40137
I20260925 18:12:19.715114 3653 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:19.715816 3653 heartbeater.cc:511] Master 127.0.205.190:40137 requested a full tablet report, sending...
I20260925 18:12:19.717753 3518 ts_manager.cc:194] Registered new tserver with Master: 3ff5b300a7d24f83848d93f0502ad32a (127.0.205.129:41021)
I20260925 18:12:19.718187 822 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.005029667s
I20260925 18:12:19.720592 3518 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37892
I20260925 18:12:19.753715 3618 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260925 18:12:19.754961 3618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:19.766963 3663 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: Bootstrap starting.
I20260925 18:12:19.771296 3663 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:19.775403 3663 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: No bootstrap required, opened a new log
I20260925 18:12:19.775803 3663 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260925 18:12:19.777724 3663 raft_consensus.cc:348] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } }
I20260925 18:12:19.778201 3663 raft_consensus.cc:374] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:19.778475 3663 raft_consensus.cc:729] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff5b300a7d24f83848d93f0502ad32a, State: Initialized, Role: FOLLOWER
I20260925 18:12:19.778998 3663 consensus_queue.cc:260] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } }
I20260925 18:12:19.779467 3663 raft_consensus.cc:388] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:19.779693 3663 raft_consensus.cc:482] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:19.779963 3663 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:19.784649 3663 raft_consensus.cc:504] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } }
I20260925 18:12:19.785288 3663 leader_election.cc:302] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ff5b300a7d24f83848d93f0502ad32a; no voters:
I20260925 18:12:19.786454 3663 leader_election.cc:288] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:19.786785 3665 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:19.788122 3665 raft_consensus.cc:686] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 1 LEADER]: Becoming Leader. State: Replica: 3ff5b300a7d24f83848d93f0502ad32a, State: Running, Role: LEADER
I20260925 18:12:19.788312 3653 heartbeater.cc:503] Master 127.0.205.190:40137 was elected leader, sending a full tablet report...
I20260925 18:12:19.788728 3663 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 18:12:19.788925 3665 consensus_queue.cc:237] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } }
I20260925 18:12:19.793195 3667 tablet_replica.cc:439] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: TxnStatusTablet state changed. Reason: New leader 3ff5b300a7d24f83848d93f0502ad32a. Latest consensus state: current_term: 1 leader_uuid: "3ff5b300a7d24f83848d93f0502ad32a" committed_config { opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } } }
I20260925 18:12:19.793926 3667 tablet_replica.cc:442] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: This TxnStatusTablet replica's current role is: LEADER
I20260925 18:12:19.794443 3666 tablet_replica.cc:439] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ff5b300a7d24f83848d93f0502ad32a" committed_config { opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } } }
I20260925 18:12:19.794962 3666 tablet_replica.cc:442] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: This TxnStatusTablet replica's current role is: LEADER
I20260925 18:12:19.795853 3670 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 18:12:19.796300 3670 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260925 18:12:19.797084 3518 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260925 18:12:19.801064 3518 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260925 18:12:19.832440 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:12:19.848286 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:19.849474 822 tablet_replica.cc:330] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: stopping tablet replica
I20260925 18:12:19.850104 822 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:19.850654 822 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:19.874099 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:12:19.885000 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:19.890754 3673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:19.891667 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:19.893594 3676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:19.899631 822 server_base.cc:1034] running on GCE node
I20260925 18:12:19.900624 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:19.900871 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:19.901021 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359939901006 us; error 0 us; skew 500 ppm
I20260925 18:12:19.901512 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:19.904265 822 webserver.cc:460] Webserver started at http://127.0.205.129:45779/ using document root <none> and password file <none>
I20260925 18:12:19.904714 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:19.904898 822 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260925 18:12:19.908438 822 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 18:12:19.911369 3681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:19.912290 822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 18:12:19.912627 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "3ff5b300a7d24f83848d93f0502ad32a"
format_stamp: "Formatted at 2026-09-25 18:12:19 on dist-test-slave-57m3"
I20260925 18:12:19.913044 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:19.930822 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:19.932114 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:19.939709 3690 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 18:12:19.946833 822 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 18:12:19.947132 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20260925 18:12:19.947456 822 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 18:12:19.953758 822 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 18:12:19.954043 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.006s sys 0.000s
I20260925 18:12:19.955037 3690 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: Bootstrap starting.
I20260925 18:12:19.992467 3690 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 18:12:19.993515 3690 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: Bootstrap complete.
I20260925 18:12:19.994179 3690 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: Time spent bootstrapping tablet: real 0.039s user 0.022s sys 0.010s
I20260925 18:12:19.996735 3690 raft_consensus.cc:348] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } }
I20260925 18:12:19.997470 3690 raft_consensus.cc:729] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff5b300a7d24f83848d93f0502ad32a, State: Initialized, Role: FOLLOWER
I20260925 18:12:19.998356 3690 consensus_queue.cc:260] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } }
I20260925 18:12:19.999034 3690 raft_consensus.cc:388] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:19.999325 3690 raft_consensus.cc:482] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:19.999647 3690 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:12:20.006682 3690 raft_consensus.cc:504] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } }
I20260925 18:12:20.007447 3690 leader_election.cc:302] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ff5b300a7d24f83848d93f0502ad32a; no voters:
I20260925 18:12:20.008916 3690 leader_election.cc:288] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 18:12:20.009279 3715 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 2 FOLLOWER]: Leader election won for term 2
I20260925 18:12:20.016698 3715 raft_consensus.cc:686] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 2 LEADER]: Becoming Leader. State: Replica: 3ff5b300a7d24f83848d93f0502ad32a, State: Running, Role: LEADER
I20260925 18:12:20.018936 3690 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a: Time spent starting tablet: real 0.024s user 0.012s sys 0.013s
I20260925 18:12:20.018607 3715 consensus_queue.cc:237] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } }
I20260925 18:12:20.021839 3717 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3ff5b300a7d24f83848d93f0502ad32a" committed_config { opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } } }
I20260925 18:12:20.022408 3717 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260925 18:12:20.023979 3721 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 18:12:20.023793 3718 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 3ff5b300a7d24f83848d93f0502ad32a. Latest consensus state: current_term: 2 leader_uuid: "3ff5b300a7d24f83848d93f0502ad32a" committed_config { opid_index: -1 peers { permanent_uuid: "3ff5b300a7d24f83848d93f0502ad32a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41021 } } }
I20260925 18:12:20.025041 3718 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260925 18:12:20.031541 3721 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 18:12:20.069453 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:41021
I20260925 18:12:20.069569 3758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:41021 every 8 connection(s)
I20260925 18:12:20.076136 3759 heartbeater.cc:347] Connected to a master server at 127.0.205.190:40137
I20260925 18:12:20.076719 3759 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:20.077252 822 tablet_server.cc:178] TabletServer@127.0.205.129:41021 shutting down...
I20260925 18:12:20.077561 3759 heartbeater.cc:511] Master 127.0.205.190:40137 requested a full tablet report, sending...
I20260925 18:12:20.080794 3518 ts_manager.cc:194] Re-registered known tserver with Master: 3ff5b300a7d24f83848d93f0502ad32a (127.0.205.129:41021)
W20260925 18:12:20.081377 3518 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260925 18:12:20.083384 3518 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37918
I20260925 18:12:20.087604 3759 heartbeater.cc:503] Master 127.0.205.190:40137 was elected leader, sending a full tablet report...
W20260925 18:12:20.090926 3518 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260925 18:12:20.103065 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:20.103871 822 tablet_replica.cc:330] stopping tablet replica
I20260925 18:12:20.104523 822 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 2 LEADER]: Raft consensus shutting down.
I20260925 18:12:20.105024 822 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 3ff5b300a7d24f83848d93f0502ad32a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:20.115459 822 tablet_server.cc:194] TabletServer@127.0.205.129:41021 shutdown complete.
I20260925 18:12:20.127728 822 master.cc:537] Master@127.0.205.190:40137 shutting down...
I20260925 18:12:20.143241 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:20.143851 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:20.144228 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P b24694fc1eff4cc292b5ab58b42b89a9: stopping tablet replica
I20260925 18:12:20.163585 822 master.cc:552] Master@127.0.205.190:40137 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (786 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20260925 18:12:20.189000 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:43251
I20260925 18:12:20.190395 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:20.195739 3765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:20.197369 3766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:20.199888 3768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:20.200734 822 server_base.cc:1034] running on GCE node
I20260925 18:12:20.201617 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:20.201856 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:20.202032 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359940202019 us; error 0 us; skew 500 ppm
I20260925 18:12:20.202702 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:20.205651 822 webserver.cc:460] Webserver started at http://127.0.205.190:40675/ using document root <none> and password file <none>
I20260925 18:12:20.206248 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:20.206461 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:20.206754 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:20.208165 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "2f9a7b1a7f884e98a86ae40a9819d265"
format_stamp: "Formatted at 2026-09-25 18:12:20 on dist-test-slave-57m3"
I20260925 18:12:20.214056 822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 18:12:20.217329 3773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:20.218106 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:12:20.218361 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "2f9a7b1a7f884e98a86ae40a9819d265"
format_stamp: "Formatted at 2026-09-25 18:12:20 on dist-test-slave-57m3"
I20260925 18:12:20.218621 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:20.235689 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:20.236701 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:20.282284 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:43251
I20260925 18:12:20.282384 3824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:43251 every 8 connection(s)
I20260925 18:12:20.286398 3825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:20.296159 3825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265: Bootstrap starting.
I20260925 18:12:20.300119 3825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:20.304050 3825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265: No bootstrap required, opened a new log
I20260925 18:12:20.305884 3825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a7b1a7f884e98a86ae40a9819d265" member_type: VOTER }
I20260925 18:12:20.306260 3825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:20.306464 3825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f9a7b1a7f884e98a86ae40a9819d265, State: Initialized, Role: FOLLOWER
I20260925 18:12:20.307029 3825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a7b1a7f884e98a86ae40a9819d265" member_type: VOTER }
I20260925 18:12:20.307430 3825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:20.307639 3825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:20.307861 3825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:20.311954 3825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a7b1a7f884e98a86ae40a9819d265" member_type: VOTER }
I20260925 18:12:20.312433 3825 leader_election.cc:302] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f9a7b1a7f884e98a86ae40a9819d265; no voters:
I20260925 18:12:20.313669 3825 leader_election.cc:288] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:20.314023 3828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:20.315593 3828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 1 LEADER]: Becoming Leader. State: Replica: 2f9a7b1a7f884e98a86ae40a9819d265, State: Running, Role: LEADER
I20260925 18:12:20.316282 3828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a7b1a7f884e98a86ae40a9819d265" member_type: VOTER }
I20260925 18:12:20.316854 3825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:12:20.319875 3829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f9a7b1a7f884e98a86ae40a9819d265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a7b1a7f884e98a86ae40a9819d265" member_type: VOTER } }
I20260925 18:12:20.319998 3830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f9a7b1a7f884e98a86ae40a9819d265. Latest consensus state: current_term: 1 leader_uuid: "2f9a7b1a7f884e98a86ae40a9819d265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9a7b1a7f884e98a86ae40a9819d265" member_type: VOTER } }
I20260925 18:12:20.320416 3829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:20.320744 3830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:20.323935 3833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:12:20.331079 3833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:12:20.342710 3833 catalog_manager.cc:1347] Generated new cluster ID: 2c3bd0bf0f3c43a4b5acb6f9a33864e1
I20260925 18:12:20.343047 3833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:12:20.346055 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:12:20.388212 3833 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:12:20.389933 3833 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:12:20.401089 3833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265: Generated new TSK 0
I20260925 18:12:20.401767 3833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:12:20.412758 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:20.419703 3846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:20.420342 3847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:20.428740 3849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:20.428810 822 server_base.cc:1034] running on GCE node
I20260925 18:12:20.430039 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:20.430294 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:20.430473 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359940430460 us; error 0 us; skew 500 ppm
I20260925 18:12:20.431159 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:20.433933 822 webserver.cc:460] Webserver started at http://127.0.205.129:34325/ using document root <none> and password file <none>
I20260925 18:12:20.434530 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:20.434752 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:20.435053 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:20.436498 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "502f804cb7ff4cba98ca8717bf2af872"
format_stamp: "Formatted at 2026-09-25 18:12:20 on dist-test-slave-57m3"
I20260925 18:12:20.442394 822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 18:12:20.446579 3854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:20.447417 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 18:12:20.447767 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "502f804cb7ff4cba98ca8717bf2af872"
format_stamp: "Formatted at 2026-09-25 18:12:20 on dist-test-slave-57m3"
I20260925 18:12:20.448120 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:20.463406 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:20.464704 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:20.476826 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:20.477140 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 18:12:20.477412 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:20.477620 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:20.544653 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:41511
I20260925 18:12:20.544710 3924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:41511 every 8 connection(s)
I20260925 18:12:20.550311 3925 heartbeater.cc:347] Connected to a master server at 127.0.205.190:43251
I20260925 18:12:20.550405 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 18:12:20.550675 3925 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:20.551334 3925 heartbeater.cc:511] Master 127.0.205.190:43251 requested a full tablet report, sending...
I20260925 18:12:20.553089 3790 ts_manager.cc:194] Registered new tserver with Master: 502f804cb7ff4cba98ca8717bf2af872 (127.0.205.129:41511)
I20260925 18:12:20.554852 3790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48214
W20260925 18:12:20.557430 3927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:20.562088 3928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:20.564229 822 server_base.cc:1034] running on GCE node
W20260925 18:12:20.565064 3930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:20.565828 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:20.566009 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:20.566176 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359940566161 us; error 0 us; skew 500 ppm
I20260925 18:12:20.566663 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:20.569098 822 webserver.cc:460] Webserver started at http://127.0.205.130:38855/ using document root <none> and password file <none>
I20260925 18:12:20.569519 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:20.569676 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:20.569912 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:20.570892 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-1-root/instance:
uuid: "d9a700ab694e423bbccacc4091b95033"
format_stamp: "Formatted at 2026-09-25 18:12:20 on dist-test-slave-57m3"
I20260925 18:12:20.575328 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 18:12:20.578617 3935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:20.579433 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 18:12:20.579762 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-1-root
uuid: "d9a700ab694e423bbccacc4091b95033"
format_stamp: "Formatted at 2026-09-25 18:12:20 on dist-test-slave-57m3"
I20260925 18:12:20.580139 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:20.607813 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:20.608857 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:20.615144 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:20.615433 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:20.615684 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:20.615823 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:20.673645 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.130:39391
I20260925 18:12:20.673708 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.130:39391 every 8 connection(s)
I20260925 18:12:20.679204 4005 heartbeater.cc:347] Connected to a master server at 127.0.205.190:43251
I20260925 18:12:20.679554 4005 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:20.680190 4005 heartbeater.cc:511] Master 127.0.205.190:43251 requested a full tablet report, sending...
I20260925 18:12:20.682071 3790 ts_manager.cc:194] Registered new tserver with Master: d9a700ab694e423bbccacc4091b95033 (127.0.205.130:39391)
I20260925 18:12:20.682703 822 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.00498171s
I20260925 18:12:20.683676 3790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48230
I20260925 18:12:20.721756 3890 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260925 18:12:20.722883 3890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:20.733852 4015 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: Bootstrap starting.
I20260925 18:12:20.738622 4015 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:20.743024 4015 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: No bootstrap required, opened a new log
I20260925 18:12:20.743482 4015 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 18:12:20.745896 4015 raft_consensus.cc:348] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } }
I20260925 18:12:20.746465 4015 raft_consensus.cc:374] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:20.746740 4015 raft_consensus.cc:729] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502f804cb7ff4cba98ca8717bf2af872, State: Initialized, Role: FOLLOWER
I20260925 18:12:20.747364 4015 consensus_queue.cc:260] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } }
I20260925 18:12:20.748005 4015 raft_consensus.cc:388] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:20.748281 4015 raft_consensus.cc:482] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:20.748613 4015 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:20.755079 4015 raft_consensus.cc:504] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } }
I20260925 18:12:20.755864 4015 leader_election.cc:302] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502f804cb7ff4cba98ca8717bf2af872; no voters:
I20260925 18:12:20.757241 4015 leader_election.cc:288] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:20.757539 4017 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:20.759240 3925 heartbeater.cc:503] Master 127.0.205.190:43251 was elected leader, sending a full tablet report...
I20260925 18:12:20.759724 4015 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: Time spent starting tablet: real 0.016s user 0.010s sys 0.007s
I20260925 18:12:20.759941 4017 raft_consensus.cc:686] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 1 LEADER]: Becoming Leader. State: Replica: 502f804cb7ff4cba98ca8717bf2af872, State: Running, Role: LEADER
I20260925 18:12:20.760607 4017 consensus_queue.cc:237] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } }
I20260925 18:12:20.763908 4019 tablet_replica.cc:439] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: TxnStatusTablet state changed. Reason: New leader 502f804cb7ff4cba98ca8717bf2af872. Latest consensus state: current_term: 1 leader_uuid: "502f804cb7ff4cba98ca8717bf2af872" committed_config { opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } } }
I20260925 18:12:20.764426 4019 tablet_replica.cc:442] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: This TxnStatusTablet replica's current role is: LEADER
I20260925 18:12:20.765055 4018 tablet_replica.cc:439] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "502f804cb7ff4cba98ca8717bf2af872" committed_config { opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } } }
I20260925 18:12:20.765594 4018 tablet_replica.cc:442] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: This TxnStatusTablet replica's current role is: LEADER
W20260925 18:12:20.766464 3790 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260925 18:12:20.771461 3790 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260925 18:12:20.773219 4021 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 18:12:20.773589 4021 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 18:12:20.832683 4026 ts_tablet_manager.cc:927] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: Initiating tablet copy from peer 502f804cb7ff4cba98ca8717bf2af872 (127.0.205.129:41511)
I20260925 18:12:20.834362 4026 tablet_copy_client.cc:316] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: tablet copy: Beginning tablet copy session from remote peer at address 127.0.205.129:41511
I20260925 18:12:20.845813 3910 tablet_copy_service.cc:140] P 502f804cb7ff4cba98ca8717bf2af872: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer d9a700ab694e423bbccacc4091b95033 ({username='slave'} at 127.0.0.1:38082)
I20260925 18:12:20.846329 3910 tablet_copy_service.cc:161] P 502f804cb7ff4cba98ca8717bf2af872: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer d9a700ab694e423bbccacc4091b95033 at {username='slave'} at 127.0.0.1:38082: session id = d9a700ab694e423bbccacc4091b95033-11111111111111111111111111111111
I20260925 18:12:20.852131 3910 tablet_copy_source_session.cc:205] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 18:12:20.854588 4026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:20.864969 4026 tablet_copy_client.cc:791] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: tablet copy: Starting download of 0 data blocks...
I20260925 18:12:20.865494 4026 tablet_copy_client.cc:655] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: tablet copy: Starting download of 1 WAL segments...
I20260925 18:12:20.868644 4026 tablet_copy_client.cc:531] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 18:12:20.874178 4026 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: Bootstrap starting.
I20260925 18:12:20.894526 4026 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 18:12:20.895267 4026 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: Bootstrap complete.
I20260925 18:12:20.895821 4026 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.002s
I20260925 18:12:20.898206 4026 raft_consensus.cc:348] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } }
I20260925 18:12:20.898748 4026 raft_consensus.cc:729] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: d9a700ab694e423bbccacc4091b95033, State: Initialized, Role: NON_PARTICIPANT
I20260925 18:12:20.899528 4026 consensus_queue.cc:260] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } }
I20260925 18:12:20.901510 4029 tablet_replica.cc:439] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } } }
I20260925 18:12:20.901849 4005 heartbeater.cc:503] Master 127.0.205.190:43251 was elected leader, sending a full tablet report...
I20260925 18:12:20.902192 4029 tablet_replica.cc:442] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20260925 18:12:20.902607 4026 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260925 18:12:20.905088 3910 tablet_copy_service.cc:342] P 502f804cb7ff4cba98ca8717bf2af872: Request end of tablet copy session d9a700ab694e423bbccacc4091b95033-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:38082
I20260925 18:12:20.905491 3910 tablet_copy_service.cc:433] P 502f804cb7ff4cba98ca8717bf2af872: ending tablet copy session d9a700ab694e423bbccacc4091b95033-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer d9a700ab694e423bbccacc4091b95033
W20260925 18:12:20.905906 3790 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260925 18:12:20.910229 3790 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260925 18:12:20.929920 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:12:20.951056 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:20.951829 822 tablet_replica.cc:330] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: stopping tablet replica
I20260925 18:12:20.952335 822 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:20.952769 822 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:20.963187 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:12:20.974614 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:20.980517 4030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:20.981936 4031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:20.984871 4033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:20.990531 822 server_base.cc:1034] running on GCE node
I20260925 18:12:20.991504 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:20.991744 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:20.991890 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359940991879 us; error 0 us; skew 500 ppm
I20260925 18:12:20.992395 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:20.995153 822 webserver.cc:460] Webserver started at http://127.0.205.129:34325/ using document root <none> and password file <none>
I20260925 18:12:20.995620 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:20.995779 822 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260925 18:12:20.999270 822 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 18:12:21.001971 4038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:21.002789 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:12:21.003126 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "502f804cb7ff4cba98ca8717bf2af872"
format_stamp: "Formatted at 2026-09-25 18:12:20 on dist-test-slave-57m3"
I20260925 18:12:21.003500 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:21.025398 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:21.026338 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:21.032915 4047 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 18:12:21.039698 822 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 18:12:21.040036 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260925 18:12:21.040364 822 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 18:12:21.046527 822 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 18:12:21.046795 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.005s sys 0.000s
I20260925 18:12:21.048074 4047 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: Bootstrap starting.
I20260925 18:12:21.085525 4047 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 18:12:21.086668 4047 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: Bootstrap complete.
I20260925 18:12:21.088366 4047 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: Time spent bootstrapping tablet: real 0.040s user 0.028s sys 0.004s
I20260925 18:12:21.091111 4047 raft_consensus.cc:348] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } }
I20260925 18:12:21.091681 4047 raft_consensus.cc:729] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502f804cb7ff4cba98ca8717bf2af872, State: Initialized, Role: FOLLOWER
I20260925 18:12:21.092322 4047 consensus_queue.cc:260] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } }
I20260925 18:12:21.094194 4047 raft_consensus.cc:388] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:21.094491 4047 raft_consensus.cc:482] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:21.094789 4047 raft_consensus.cc:3037] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:12:21.100082 4047 raft_consensus.cc:504] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } }
I20260925 18:12:21.100903 4047 leader_election.cc:302] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502f804cb7ff4cba98ca8717bf2af872; no voters:
I20260925 18:12:21.103518 4047 leader_election.cc:288] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 18:12:21.103816 4060 raft_consensus.cc:2781] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 18:12:21.112494 4060 raft_consensus.cc:686] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 2 LEADER]: Becoming Leader. State: Replica: 502f804cb7ff4cba98ca8717bf2af872, State: Running, Role: LEADER
I20260925 18:12:21.113431 4060 consensus_queue.cc:237] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } }
I20260925 18:12:21.114235 4047 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872: Time spent starting tablet: real 0.026s user 0.016s sys 0.008s
I20260925 18:12:21.116060 4063 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 502f804cb7ff4cba98ca8717bf2af872. Latest consensus state: current_term: 2 leader_uuid: "502f804cb7ff4cba98ca8717bf2af872" committed_config { opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } } }
I20260925 18:12:21.116641 4063 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260925 18:12:21.116454 4064 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "502f804cb7ff4cba98ca8717bf2af872" committed_config { opid_index: -1 peers { permanent_uuid: "502f804cb7ff4cba98ca8717bf2af872" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41511 } } }
I20260925 18:12:21.117405 4064 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260925 18:12:21.122282 4067 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 18:12:21.122677 4067 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 18:12:21.167026 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:41511
I20260925 18:12:21.167133 4115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:41511 every 8 connection(s)
I20260925 18:12:21.172665 4116 heartbeater.cc:347] Connected to a master server at 127.0.205.190:43251
I20260925 18:12:21.173161 4116 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:21.174006 4116 heartbeater.cc:511] Master 127.0.205.190:43251 requested a full tablet report, sending...
I20260925 18:12:21.174425 822 tablet_server.cc:178] TabletServer@127.0.205.129:41511 shutting down...
I20260925 18:12:21.178797 3790 ts_manager.cc:194] Re-registered known tserver with Master: 502f804cb7ff4cba98ca8717bf2af872 (127.0.205.129:41511)
W20260925 18:12:21.180676 3790 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260925 18:12:21.183274 3790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48260
I20260925 18:12:21.189517 4116 heartbeater.cc:503] Master 127.0.205.190:43251 was elected leader, sending a full tablet report...
W20260925 18:12:21.192965 3790 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260925 18:12:21.197568 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:21.198386 822 tablet_replica.cc:330] stopping tablet replica
I20260925 18:12:21.198871 822 raft_consensus.cc:2229] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 2 LEADER]: Raft consensus shutting down.
I20260925 18:12:21.199285 822 raft_consensus.cc:2258] T 11111111111111111111111111111111 P 502f804cb7ff4cba98ca8717bf2af872 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:21.210470 822 tablet_server.cc:194] TabletServer@127.0.205.129:41511 shutdown complete.
I20260925 18:12:21.220686 822 tablet_server.cc:178] TabletServer@127.0.205.130:0 shutting down...
I20260925 18:12:21.236016 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:21.236719 822 tablet_replica.cc:330] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033: stopping tablet replica
I20260925 18:12:21.237283 822 raft_consensus.cc:2229] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 18:12:21.237685 822 raft_consensus.cc:2258] T 11111111111111111111111111111111 P d9a700ab694e423bbccacc4091b95033 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 18:12:21.255934 822 tablet_server.cc:194] TabletServer@127.0.205.130:0 shutdown complete.
I20260925 18:12:21.267759 822 master.cc:537] Master@127.0.205.190:43251 shutting down...
I20260925 18:12:21.286345 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:21.286819 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:21.287081 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2f9a7b1a7f884e98a86ae40a9819d265: stopping tablet replica
I20260925 18:12:21.295107 822 master.cc:552] Master@127.0.205.190:43251 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1132 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20260925 18:12:21.320055 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:40141
I20260925 18:12:21.321110 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:21.326159 4122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:21.327665 4123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:21.330121 4125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:21.331033 822 server_base.cc:1034] running on GCE node
I20260925 18:12:21.331717 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:21.331880 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:21.332011 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359941331997 us; error 0 us; skew 500 ppm
I20260925 18:12:21.332490 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:21.334897 822 webserver.cc:460] Webserver started at http://127.0.205.190:44247/ using document root <none> and password file <none>
I20260925 18:12:21.335353 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:21.335516 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:21.335749 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:21.336728 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "3f920aee4ce549cb8d8005b927e764a7"
format_stamp: "Formatted at 2026-09-25 18:12:21 on dist-test-slave-57m3"
I20260925 18:12:21.341070 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 18:12:21.344421 4130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:21.345230 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:12:21.345490 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "3f920aee4ce549cb8d8005b927e764a7"
format_stamp: "Formatted at 2026-09-25 18:12:21 on dist-test-slave-57m3"
I20260925 18:12:21.345757 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:21.401566 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:21.402539 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:21.447049 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:40141
I20260925 18:12:21.447212 4181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:40141 every 8 connection(s)
I20260925 18:12:21.451205 4182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:21.465389 4182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7: Bootstrap starting.
I20260925 18:12:21.470925 4182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:21.475505 4182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7: No bootstrap required, opened a new log
I20260925 18:12:21.477947 4182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f920aee4ce549cb8d8005b927e764a7" member_type: VOTER }
I20260925 18:12:21.478451 4182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:21.478715 4182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f920aee4ce549cb8d8005b927e764a7, State: Initialized, Role: FOLLOWER
I20260925 18:12:21.479326 4182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f920aee4ce549cb8d8005b927e764a7" member_type: VOTER }
I20260925 18:12:21.479899 4182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:21.480172 4182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:21.480469 4182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:21.486773 4182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f920aee4ce549cb8d8005b927e764a7" member_type: VOTER }
I20260925 18:12:21.487460 4182 leader_election.cc:302] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f920aee4ce549cb8d8005b927e764a7; no voters:
I20260925 18:12:21.488722 4182 leader_election.cc:288] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:21.489032 4185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:21.490813 4185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 1 LEADER]: Becoming Leader. State: Replica: 3f920aee4ce549cb8d8005b927e764a7, State: Running, Role: LEADER
I20260925 18:12:21.491465 4185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f920aee4ce549cb8d8005b927e764a7" member_type: VOTER }
I20260925 18:12:21.491855 4182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:12:21.493716 4186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f920aee4ce549cb8d8005b927e764a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f920aee4ce549cb8d8005b927e764a7" member_type: VOTER } }
I20260925 18:12:21.494112 4186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:21.494812 4187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f920aee4ce549cb8d8005b927e764a7. Latest consensus state: current_term: 1 leader_uuid: "3f920aee4ce549cb8d8005b927e764a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f920aee4ce549cb8d8005b927e764a7" member_type: VOTER } }
I20260925 18:12:21.495359 4187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:21.510630 4190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:12:21.515796 4190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:12:21.519364 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:12:21.525573 4190 catalog_manager.cc:1347] Generated new cluster ID: 1b3db64f33cc4788816fe6c99a1ea670
I20260925 18:12:21.525846 4190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:12:21.543905 4190 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:12:21.545642 4190 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:12:21.558574 4190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7: Generated new TSK 0
I20260925 18:12:21.559212 4190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:12:21.588117 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:21.593964 4203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:21.597196 4204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:21.597344 822 server_base.cc:1034] running on GCE node
W20260925 18:12:21.598618 4206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:21.599535 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:21.599720 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:21.599886 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359941599865 us; error 0 us; skew 500 ppm
I20260925 18:12:21.600441 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:21.603183 822 webserver.cc:460] Webserver started at http://127.0.205.129:44515/ using document root <none> and password file <none>
I20260925 18:12:21.603655 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:21.603818 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:21.604055 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:21.605147 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "e53a8a77feab4e43a0361ae21994397a"
format_stamp: "Formatted at 2026-09-25 18:12:21 on dist-test-slave-57m3"
I20260925 18:12:21.609588 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 18:12:21.612754 4211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:21.613612 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:12:21.613868 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "e53a8a77feab4e43a0361ae21994397a"
format_stamp: "Formatted at 2026-09-25 18:12:21 on dist-test-slave-57m3"
I20260925 18:12:21.614122 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:21.631160 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:21.632197 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:21.644117 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:21.644472 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:21.644789 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:21.645008 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 18:12:21.711259 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:40775
I20260925 18:12:21.711338 4281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:40775 every 8 connection(s)
I20260925 18:12:21.716209 4282 heartbeater.cc:347] Connected to a master server at 127.0.205.190:40141
I20260925 18:12:21.716545 4282 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:21.717247 4282 heartbeater.cc:511] Master 127.0.205.190:40141 requested a full tablet report, sending...
I20260925 18:12:21.719105 4147 ts_manager.cc:194] Registered new tserver with Master: e53a8a77feab4e43a0361ae21994397a (127.0.205.129:40775)
I20260925 18:12:21.719374 822 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004862285s
I20260925 18:12:21.721055 4147 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59150
I20260925 18:12:21.758281 4247 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260925 18:12:21.759455 4247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:21.770961 4292 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a: Bootstrap starting.
I20260925 18:12:21.775094 4292 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:21.778976 4292 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a: No bootstrap required, opened a new log
I20260925 18:12:21.779347 4292 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260925 18:12:21.781064 4292 raft_consensus.cc:348] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "e53a8a77feab4e43a0361ae21994397a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40775 } }
I20260925 18:12:21.781447 4292 raft_consensus.cc:374] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:21.781642 4292 raft_consensus.cc:729] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53a8a77feab4e43a0361ae21994397a, State: Initialized, Role: FOLLOWER
I20260925 18:12:21.782132 4292 consensus_queue.cc:260] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e53a8a77feab4e43a0361ae21994397a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40775 } }
I20260925 18:12:21.782573 4292 raft_consensus.cc:388] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:21.782768 4292 raft_consensus.cc:482] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:21.783015 4292 raft_consensus.cc:3037] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:21.787240 4292 raft_consensus.cc:504] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "e53a8a77feab4e43a0361ae21994397a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40775 } }
I20260925 18:12:21.787756 4292 leader_election.cc:302] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53a8a77feab4e43a0361ae21994397a; no voters:
I20260925 18:12:21.789049 4292 leader_election.cc:288] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:21.789397 4294 raft_consensus.cc:2781] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:21.790894 4282 heartbeater.cc:503] Master 127.0.205.190:40141 was elected leader, sending a full tablet report...
I20260925 18:12:21.791397 4292 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260925 18:12:21.791553 4294 raft_consensus.cc:686] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 1 LEADER]: Becoming Leader. State: Replica: e53a8a77feab4e43a0361ae21994397a, State: Running, Role: LEADER
I20260925 18:12:21.792193 4294 consensus_queue.cc:237] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "e53a8a77feab4e43a0361ae21994397a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40775 } }
I20260925 18:12:21.794941 4295 tablet_replica.cc:439] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53a8a77feab4e43a0361ae21994397a" committed_config { opid_index: -1 peers { permanent_uuid: "e53a8a77feab4e43a0361ae21994397a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40775 } } }
I20260925 18:12:21.795488 4295 tablet_replica.cc:442] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a: This TxnStatusTablet replica's current role is: LEADER
W20260925 18:12:21.797773 4147 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260925 18:12:21.799281 4298 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 18:12:21.799319 4296 tablet_replica.cc:439] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a: TxnStatusTablet state changed. Reason: New leader e53a8a77feab4e43a0361ae21994397a. Latest consensus state: current_term: 1 leader_uuid: "e53a8a77feab4e43a0361ae21994397a" committed_config { opid_index: -1 peers { permanent_uuid: "e53a8a77feab4e43a0361ae21994397a" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40775 } } }
I20260925 18:12:21.799856 4298 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 18:12:21.800063 4296 tablet_replica.cc:442] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a: This TxnStatusTablet replica's current role is: LEADER
W20260925 18:12:21.802198 4147 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260925 18:12:21.860307 4238 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260925 18:12:21.860608 4238 txn_status_manager.cc:243] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260925 18:12:21.860909 4238 txn_status_manager.cc:205] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20260925 18:12:21.869786 4238 txn_status_manager.cc:336] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260925 18:12:21.911731 4238 txn_status_manager.cc:243] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20260925 18:12:21.912081 4238 txn_status_manager.cc:205] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20260925 18:12:21.920284 4238 txn_status_manager.cc:336] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20260925 18:12:21.934168 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:55046
W20260925 18:12:21.936396 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:55046
W20260925 18:12:21.938472 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:55046
W20260925 18:12:21.940550 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:55046
W20260925 18:12:21.943315 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:55046
W20260925 18:12:21.945900 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:55046
W20260925 18:12:21.959293 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:55054
W20260925 18:12:21.961167 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:55054
W20260925 18:12:21.962815 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:55054
W20260925 18:12:21.964462 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:55054
W20260925 18:12:21.966282 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:55054
W20260925 18:12:21.967912 4247 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:55054
I20260925 18:12:21.971457 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:12:21.994745 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:21.995513 822 tablet_replica.cc:330] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a: stopping tablet replica
I20260925 18:12:21.995990 822 raft_consensus.cc:2229] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:21.996412 822 raft_consensus.cc:2258] T 11111111111111111111111111111111 P e53a8a77feab4e43a0361ae21994397a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:22.007803 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:12:22.017904 822 master.cc:537] Master@127.0.205.190:40141 shutting down...
I20260925 18:12:22.031714 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:22.032202 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:22.032593 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3f920aee4ce549cb8d8005b927e764a7: stopping tablet replica
I20260925 18:12:22.049916 822 master.cc:552] Master@127.0.205.190:40141 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (751 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20260925 18:12:22.071059 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:44491
I20260925 18:12:22.072022 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:22.076599 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:22.077587 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:22.078567 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:22.079056 822 server_base.cc:1034] running on GCE node
I20260925 18:12:22.079682 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:22.079842 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:22.079979 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359942079966 us; error 0 us; skew 500 ppm
I20260925 18:12:22.080472 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:22.082607 822 webserver.cc:460] Webserver started at http://127.0.205.190:46345/ using document root <none> and password file <none>
I20260925 18:12:22.083033 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:22.083199 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:22.083428 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:22.084434 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "f30992c62c6c43e697fff1df86019aeb"
format_stamp: "Formatted at 2026-09-25 18:12:22 on dist-test-slave-57m3"
I20260925 18:12:22.088706 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 18:12:22.091811 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:22.092483 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 18:12:22.092741 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "f30992c62c6c43e697fff1df86019aeb"
format_stamp: "Formatted at 2026-09-25 18:12:22 on dist-test-slave-57m3"
I20260925 18:12:22.093096 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:22.110539 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:22.111466 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:22.143508 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:44491
I20260925 18:12:22.143596 4366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:44491 every 8 connection(s)
I20260925 18:12:22.146988 4367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:22.156221 4367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb: Bootstrap starting.
I20260925 18:12:22.160318 4367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:22.164048 4367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb: No bootstrap required, opened a new log
I20260925 18:12:22.165855 4367 raft_consensus.cc:348] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30992c62c6c43e697fff1df86019aeb" member_type: VOTER }
I20260925 18:12:22.166213 4367 raft_consensus.cc:374] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:22.166427 4367 raft_consensus.cc:729] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30992c62c6c43e697fff1df86019aeb, State: Initialized, Role: FOLLOWER
I20260925 18:12:22.166981 4367 consensus_queue.cc:260] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30992c62c6c43e697fff1df86019aeb" member_type: VOTER }
I20260925 18:12:22.167400 4367 raft_consensus.cc:388] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:22.167637 4367 raft_consensus.cc:482] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:22.167871 4367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:22.171975 4367 raft_consensus.cc:504] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30992c62c6c43e697fff1df86019aeb" member_type: VOTER }
I20260925 18:12:22.172448 4367 leader_election.cc:302] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f30992c62c6c43e697fff1df86019aeb; no voters:
I20260925 18:12:22.173501 4367 leader_election.cc:288] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:22.173786 4370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:22.174940 4370 raft_consensus.cc:686] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 1 LEADER]: Becoming Leader. State: Replica: f30992c62c6c43e697fff1df86019aeb, State: Running, Role: LEADER
I20260925 18:12:22.175545 4370 consensus_queue.cc:237] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30992c62c6c43e697fff1df86019aeb" member_type: VOTER }
I20260925 18:12:22.176127 4367 sys_catalog.cc:564] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:12:22.178336 4372 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f30992c62c6c43e697fff1df86019aeb. Latest consensus state: current_term: 1 leader_uuid: "f30992c62c6c43e697fff1df86019aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30992c62c6c43e697fff1df86019aeb" member_type: VOTER } }
I20260925 18:12:22.178297 4371 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f30992c62c6c43e697fff1df86019aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30992c62c6c43e697fff1df86019aeb" member_type: VOTER } }
I20260925 18:12:22.178984 4372 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:22.179111 4371 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:22.182909 4375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:12:22.187235 4375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:12:22.189217 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:12:22.195734 4375 catalog_manager.cc:1347] Generated new cluster ID: 4706c4a075a34fd699892731879c09dc
I20260925 18:12:22.196027 4375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:12:22.236729 4375 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:12:22.238627 4375 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:12:22.253674 4375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb: Generated new TSK 0
I20260925 18:12:22.254417 4375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:12:22.274214 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:22.281709 4389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:22.283275 4390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:22.283834 4392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:22.283957 822 server_base.cc:1034] running on GCE node
I20260925 18:12:22.285066 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:22.285249 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:22.285359 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359942285351 us; error 0 us; skew 500 ppm
I20260925 18:12:22.285815 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:22.288229 822 webserver.cc:460] Webserver started at http://127.0.205.129:40519/ using document root <none> and password file <none>
I20260925 18:12:22.288894 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:22.289127 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:22.289518 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:22.291479 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "b33813122f5e4e66a8b6d2d544bdcf28"
format_stamp: "Formatted at 2026-09-25 18:12:22 on dist-test-slave-57m3"
I20260925 18:12:22.297612 822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 18:12:22.301048 4397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:22.301821 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:12:22.302078 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "b33813122f5e4e66a8b6d2d544bdcf28"
format_stamp: "Formatted at 2026-09-25 18:12:22 on dist-test-slave-57m3"
I20260925 18:12:22.302358 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:22.336295 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:22.337568 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:22.342793 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:22.343139 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:22.343461 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:22.343699 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:22.398065 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:45339
I20260925 18:12:22.398205 4467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:45339 every 8 connection(s)
I20260925 18:12:22.402714 4468 heartbeater.cc:347] Connected to a master server at 127.0.205.190:44491
I20260925 18:12:22.403090 4468 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:22.403807 4468 heartbeater.cc:511] Master 127.0.205.190:44491 requested a full tablet report, sending...
I20260925 18:12:22.405783 4332 ts_manager.cc:194] Registered new tserver with Master: b33813122f5e4e66a8b6d2d544bdcf28 (127.0.205.129:45339)
I20260925 18:12:22.406073 822 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00476392s
I20260925 18:12:22.407730 4332 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42706
I20260925 18:12:22.441519 4433 tablet_service.cc:1467] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260925 18:12:22.442737 4433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:22.455955 4478 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28: Bootstrap starting.
I20260925 18:12:22.460546 4478 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:22.464581 4478 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28: No bootstrap required, opened a new log
I20260925 18:12:22.464982 4478 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260925 18:12:22.466840 4478 raft_consensus.cc:348] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45339 } }
I20260925 18:12:22.467434 4478 raft_consensus.cc:374] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:22.467674 4478 raft_consensus.cc:729] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33813122f5e4e66a8b6d2d544bdcf28, State: Initialized, Role: FOLLOWER
I20260925 18:12:22.468207 4478 consensus_queue.cc:260] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45339 } }
I20260925 18:12:22.468680 4478 raft_consensus.cc:388] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:22.468923 4478 raft_consensus.cc:482] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:22.469170 4478 raft_consensus.cc:3037] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:22.474337 4478 raft_consensus.cc:504] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45339 } }
I20260925 18:12:22.474861 4478 leader_election.cc:302] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b33813122f5e4e66a8b6d2d544bdcf28; no voters:
I20260925 18:12:22.475967 4478 leader_election.cc:288] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:22.476406 4480 raft_consensus.cc:2781] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:22.477896 4468 heartbeater.cc:503] Master 127.0.205.190:44491 was elected leader, sending a full tablet report...
I20260925 18:12:22.478197 4478 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260925 18:12:22.478235 4480 raft_consensus.cc:686] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 1 LEADER]: Becoming Leader. State: Replica: b33813122f5e4e66a8b6d2d544bdcf28, State: Running, Role: LEADER
I20260925 18:12:22.479035 4480 consensus_queue.cc:237] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45339 } }
I20260925 18:12:22.485152 4482 tablet_replica.cc:439] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28: TxnStatusTablet state changed. Reason: New leader b33813122f5e4e66a8b6d2d544bdcf28. Latest consensus state: current_term: 1 leader_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" committed_config { opid_index: -1 peers { permanent_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45339 } } }
I20260925 18:12:22.485628 4482 tablet_replica.cc:442] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28: This TxnStatusTablet replica's current role is: LEADER
I20260925 18:12:22.486346 4481 tablet_replica.cc:439] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" committed_config { opid_index: -1 peers { permanent_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45339 } } }
I20260925 18:12:22.486896 4481 tablet_replica.cc:442] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28: This TxnStatusTablet replica's current role is: LEADER
I20260925 18:12:22.487126 4485 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 18:12:22.487565 4485 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260925 18:12:22.488616 4332 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
W20260925 18:12:22.492195 4332 catalog_manager.cc:5328] Ignoring report from unknown tablet 11111111111111111111111111111111
I20260925 18:12:22.514528 4433 tablet_service.cc:1467] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20260925 18:12:22.515591 4433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:22.525426 4478 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28: Bootstrap starting.
I20260925 18:12:22.529721 4478 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:22.533747 4478 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28: No bootstrap required, opened a new log
I20260925 18:12:22.534109 4478 ts_tablet_manager.cc:1397] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260925 18:12:22.535969 4478 raft_consensus.cc:348] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45339 } }
I20260925 18:12:22.536368 4478 raft_consensus.cc:374] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:22.536578 4478 raft_consensus.cc:729] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33813122f5e4e66a8b6d2d544bdcf28, State: Initialized, Role: FOLLOWER
I20260925 18:12:22.537046 4478 consensus_queue.cc:260] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45339 } }
I20260925 18:12:22.537477 4478 raft_consensus.cc:388] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:22.537685 4478 raft_consensus.cc:482] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:22.537914 4478 raft_consensus.cc:3037] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:22.542447 4478 raft_consensus.cc:504] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45339 } }
I20260925 18:12:22.542949 4478 leader_election.cc:302] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b33813122f5e4e66a8b6d2d544bdcf28; no voters:
I20260925 18:12:22.543414 4478 leader_election.cc:288] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:22.543622 4481 raft_consensus.cc:2781] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:22.544055 4481 raft_consensus.cc:686] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 1 LEADER]: Becoming Leader. State: Replica: b33813122f5e4e66a8b6d2d544bdcf28, State: Running, Role: LEADER
I20260925 18:12:22.544721 4478 ts_tablet_manager.cc:1428] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260925 18:12:22.544694 4481 consensus_queue.cc:237] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "b33813122f5e4e66a8b6d2d544bdcf28" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 45339 } }
W20260925 18:12:22.550870 4332 catalog_manager.cc:5328] Ignoring report from unknown tablet 22222222222222222222222222222222
W20260925 18:12:22.554541 4332 catalog_manager.cc:5328] Ignoring report from unknown tablet 22222222222222222222222222222222
I20260925 18:12:22.574432 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:12:22.597631 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:22.598440 822 tablet_replica.cc:330] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28: stopping tablet replica
I20260925 18:12:22.599112 822 raft_consensus.cc:2229] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:22.599535 822 raft_consensus.cc:2258] T 22222222222222222222222222222222 P b33813122f5e4e66a8b6d2d544bdcf28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:22.601914 822 tablet_replica.cc:330] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28: stopping tablet replica
I20260925 18:12:22.602353 822 raft_consensus.cc:2229] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:22.602793 822 raft_consensus.cc:2258] T 11111111111111111111111111111111 P b33813122f5e4e66a8b6d2d544bdcf28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:22.623337 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:12:22.633862 822 master.cc:537] Master@127.0.205.190:44491 shutting down...
I20260925 18:12:22.651849 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:22.652356 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:22.652773 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P f30992c62c6c43e697fff1df86019aeb: stopping tablet replica
I20260925 18:12:22.672191 822 master.cc:552] Master@127.0.205.190:44491 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (629 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3299 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20260925 18:12:22.701081 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:44355
I20260925 18:12:22.702101 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:22.706935 4494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:22.708647 4495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:22.708966 4497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:22.709398 822 server_base.cc:1034] running on GCE node
I20260925 18:12:22.710076 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:22.710237 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:22.710397 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359942710375 us; error 0 us; skew 500 ppm
I20260925 18:12:22.710894 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:22.713105 822 webserver.cc:460] Webserver started at http://127.0.205.190:41755/ using document root <none> and password file <none>
I20260925 18:12:22.713547 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:22.713706 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:22.713938 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:22.714993 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "d45656418a9342d1a2f71f5d804d8444"
format_stamp: "Formatted at 2026-09-25 18:12:22 on dist-test-slave-57m3"
I20260925 18:12:22.719213 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 18:12:22.722524 4502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:22.723212 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 18:12:22.723479 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "d45656418a9342d1a2f71f5d804d8444"
format_stamp: "Formatted at 2026-09-25 18:12:22 on dist-test-slave-57m3"
I20260925 18:12:22.723747 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:22.750603 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:22.751765 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:22.787335 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:44355
I20260925 18:12:22.787432 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:44355 every 8 connection(s)
I20260925 18:12:22.790984 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:22.802145 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444: Bootstrap starting.
I20260925 18:12:22.806623 4554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:22.810689 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444: No bootstrap required, opened a new log
I20260925 18:12:22.812621 4554 raft_consensus.cc:348] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45656418a9342d1a2f71f5d804d8444" member_type: VOTER }
I20260925 18:12:22.813056 4554 raft_consensus.cc:374] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:22.813241 4554 raft_consensus.cc:729] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45656418a9342d1a2f71f5d804d8444, State: Initialized, Role: FOLLOWER
I20260925 18:12:22.813791 4554 consensus_queue.cc:260] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45656418a9342d1a2f71f5d804d8444" member_type: VOTER }
I20260925 18:12:22.814198 4554 raft_consensus.cc:388] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:22.814373 4554 raft_consensus.cc:482] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:22.814595 4554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:22.818953 4554 raft_consensus.cc:504] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45656418a9342d1a2f71f5d804d8444" member_type: VOTER }
I20260925 18:12:22.819427 4554 leader_election.cc:302] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d45656418a9342d1a2f71f5d804d8444; no voters:
I20260925 18:12:22.820477 4554 leader_election.cc:288] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:22.820900 4557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:22.822127 4557 raft_consensus.cc:686] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 1 LEADER]: Becoming Leader. State: Replica: d45656418a9342d1a2f71f5d804d8444, State: Running, Role: LEADER
I20260925 18:12:22.822672 4557 consensus_queue.cc:237] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45656418a9342d1a2f71f5d804d8444" member_type: VOTER }
I20260925 18:12:22.823237 4554 sys_catalog.cc:564] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:12:22.825429 4559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45656418a9342d1a2f71f5d804d8444. Latest consensus state: current_term: 1 leader_uuid: "d45656418a9342d1a2f71f5d804d8444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45656418a9342d1a2f71f5d804d8444" member_type: VOTER } }
I20260925 18:12:22.825510 4558 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d45656418a9342d1a2f71f5d804d8444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45656418a9342d1a2f71f5d804d8444" member_type: VOTER } }
I20260925 18:12:22.826196 4558 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:22.826171 4559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:22.828891 4562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:12:22.835770 4562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:12:22.837523 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:12:22.844414 4562 catalog_manager.cc:1347] Generated new cluster ID: 12e93f392d0e4e1fbdacc04cc9f97d33
I20260925 18:12:22.844739 4562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:12:22.862831 4562 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:12:22.864537 4562 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:12:22.881897 4562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444: Generated new TSK 0
I20260925 18:12:22.882594 4562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:12:22.903847 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:22.909299 4575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:22.910454 4576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:22.912871 4578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:22.913678 822 server_base.cc:1034] running on GCE node
I20260925 18:12:22.914368 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:22.914561 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:22.914702 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359942914686 us; error 0 us; skew 500 ppm
I20260925 18:12:22.915195 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:22.917388 822 webserver.cc:460] Webserver started at http://127.0.205.129:32843/ using document root <none> and password file <none>
I20260925 18:12:22.917822 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:22.917980 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:22.918205 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:22.919239 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "63c5ea5ac9074f0cb2459f3bd81016a7"
format_stamp: "Formatted at 2026-09-25 18:12:22 on dist-test-slave-57m3"
I20260925 18:12:22.923545 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 18:12:22.926836 4583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:22.927507 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:12:22.927763 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "63c5ea5ac9074f0cb2459f3bd81016a7"
format_stamp: "Formatted at 2026-09-25 18:12:22 on dist-test-slave-57m3"
I20260925 18:12:22.928028 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:22.940755 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:22.941830 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:22.943212 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:12:22.945425 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:22.945606 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:22.945820 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:22.945967 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:22.981835 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:40949
I20260925 18:12:22.981937 4645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:40949 every 8 connection(s)
I20260925 18:12:22.986430 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:22.992992 4650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:22.994889 4651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:22.999495 822 server_base.cc:1034] running on GCE node
W20260925 18:12:23.000437 4653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:23.000643 4646 heartbeater.cc:347] Connected to a master server at 127.0.205.190:44355
I20260925 18:12:23.001108 4646 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:23.001397 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:23.001623 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:23.001739 4646 heartbeater.cc:511] Master 127.0.205.190:44355 requested a full tablet report, sending...
I20260925 18:12:23.001772 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359943001760 us; error 0 us; skew 500 ppm
I20260925 18:12:23.002599 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:23.003924 4519 ts_manager.cc:194] Registered new tserver with Master: 63c5ea5ac9074f0cb2459f3bd81016a7 (127.0.205.129:40949)
I20260925 18:12:23.005281 822 webserver.cc:460] Webserver started at http://127.0.205.130:40097/ using document root <none> and password file <none>
I20260925 18:12:23.005744 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:23.005910 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:23.006140 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:23.006309 4519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60102
I20260925 18:12:23.007356 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-1-root/instance:
uuid: "78827e238b574577a7ae31cbdb1b9c66"
format_stamp: "Formatted at 2026-09-25 18:12:23 on dist-test-slave-57m3"
I20260925 18:12:23.011652 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 18:12:23.014874 4658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:23.015604 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:12:23.015870 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-1-root
uuid: "78827e238b574577a7ae31cbdb1b9c66"
format_stamp: "Formatted at 2026-09-25 18:12:23 on dist-test-slave-57m3"
I20260925 18:12:23.016166 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:23.031836 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:23.033010 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:23.034343 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:12:23.036470 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:23.036651 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:23.036888 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:23.037029 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:23.072968 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.130:44467
I20260925 18:12:23.073071 4720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.130:44467 every 8 connection(s)
I20260925 18:12:23.077548 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:23.085345 4725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:23.086225 4724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:23.087194 822 server_base.cc:1034] running on GCE node
W20260925 18:12:23.087414 4727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:23.088370 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:23.088568 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:23.088771 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359943088755 us; error 0 us; skew 500 ppm
I20260925 18:12:23.089480 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:23.090690 4721 heartbeater.cc:347] Connected to a master server at 127.0.205.190:44355
I20260925 18:12:23.091109 4721 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:23.091753 4721 heartbeater.cc:511] Master 127.0.205.190:44355 requested a full tablet report, sending...
I20260925 18:12:23.092415 822 webserver.cc:460] Webserver started at http://127.0.205.131:42379/ using document root <none> and password file <none>
I20260925 18:12:23.093024 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:23.093195 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:23.093425 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:23.093704 4519 ts_manager.cc:194] Registered new tserver with Master: 78827e238b574577a7ae31cbdb1b9c66 (127.0.205.130:44467)
I20260925 18:12:23.094538 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-2-root/instance:
uuid: "298670e239e94e53b0fccb47cff216b8"
format_stamp: "Formatted at 2026-09-25 18:12:23 on dist-test-slave-57m3"
I20260925 18:12:23.095300 4519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60116
I20260925 18:12:23.099114 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 18:12:23.102346 4732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:23.103137 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 18:12:23.103391 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-2-root
uuid: "298670e239e94e53b0fccb47cff216b8"
format_stamp: "Formatted at 2026-09-25 18:12:23 on dist-test-slave-57m3"
I20260925 18:12:23.103628 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:23.119783 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:23.120908 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:23.122277 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:12:23.124513 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:23.124701 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:23.124928 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:23.125072 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:23.162987 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.131:40455
I20260925 18:12:23.163075 4794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.131:40455 every 8 connection(s)
I20260925 18:12:23.177723 4795 heartbeater.cc:347] Connected to a master server at 127.0.205.190:44355
I20260925 18:12:23.178081 4795 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:23.178704 4795 heartbeater.cc:511] Master 127.0.205.190:44355 requested a full tablet report, sending...
I20260925 18:12:23.180621 4519 ts_manager.cc:194] Registered new tserver with Master: 298670e239e94e53b0fccb47cff216b8 (127.0.205.131:40455)
I20260925 18:12:23.181344 822 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015384764s
I20260925 18:12:23.182200 4519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60122
I20260925 18:12:23.201884 822 test_util.cc:268] Using random seed: -1469103034
I20260925 18:12:23.224980 4519 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60126:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 18:12:23.268602 4760 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf0c60abf94c79a10f61aef0136864 (DEFAULT_TABLE table=test-workload [id=42f5d6a142674df086c6e6d0079d0193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:12:23.269937 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf0c60abf94c79a10f61aef0136864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:23.280534 4611 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf0c60abf94c79a10f61aef0136864 (DEFAULT_TABLE table=test-workload [id=42f5d6a142674df086c6e6d0079d0193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:12:23.280624 4686 tablet_service.cc:1467] Processing CreateTablet for tablet a3bf0c60abf94c79a10f61aef0136864 (DEFAULT_TABLE table=test-workload [id=42f5d6a142674df086c6e6d0079d0193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:12:23.282047 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf0c60abf94c79a10f61aef0136864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:23.282051 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf0c60abf94c79a10f61aef0136864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:23.284171 4815 tablet_bootstrap.cc:492] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Bootstrap starting.
I20260925 18:12:23.289186 4815 tablet_bootstrap.cc:654] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:23.302382 4815 tablet_bootstrap.cc:492] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: No bootstrap required, opened a new log
I20260925 18:12:23.302922 4815 ts_tablet_manager.cc:1397] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260925 18:12:23.305766 4815 raft_consensus.cc:348] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } }
I20260925 18:12:23.306545 4815 raft_consensus.cc:374] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:23.306841 4815 raft_consensus.cc:729] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298670e239e94e53b0fccb47cff216b8, State: Initialized, Role: FOLLOWER
I20260925 18:12:23.307391 4817 tablet_bootstrap.cc:492] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66: Bootstrap starting.
I20260925 18:12:23.307785 4815 consensus_queue.cc:260] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } }
I20260925 18:12:23.309098 4818 tablet_bootstrap.cc:492] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7: Bootstrap starting.
I20260925 18:12:23.310881 4795 heartbeater.cc:503] Master 127.0.205.190:44355 was elected leader, sending a full tablet report...
I20260925 18:12:23.311055 4815 ts_tablet_manager.cc:1428] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260925 18:12:23.317054 4817 tablet_bootstrap.cc:654] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:23.317176 4818 tablet_bootstrap.cc:654] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:23.322067 4817 tablet_bootstrap.cc:492] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66: No bootstrap required, opened a new log
I20260925 18:12:23.322491 4817 ts_tablet_manager.cc:1397] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
I20260925 18:12:23.322508 4818 tablet_bootstrap.cc:492] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7: No bootstrap required, opened a new log
I20260925 18:12:23.323112 4818 ts_tablet_manager.cc:1397] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260925 18:12:23.325053 4817 raft_consensus.cc:348] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } }
I20260925 18:12:23.325758 4817 raft_consensus.cc:374] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:23.326036 4817 raft_consensus.cc:729] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78827e238b574577a7ae31cbdb1b9c66, State: Initialized, Role: FOLLOWER
I20260925 18:12:23.325783 4818 raft_consensus.cc:348] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } }
I20260925 18:12:23.326488 4818 raft_consensus.cc:374] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:23.326694 4818 raft_consensus.cc:729] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c5ea5ac9074f0cb2459f3bd81016a7, State: Initialized, Role: FOLLOWER
I20260925 18:12:23.326749 4817 consensus_queue.cc:260] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } }
I20260925 18:12:23.327214 4818 consensus_queue.cc:260] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } }
I20260925 18:12:23.329074 4721 heartbeater.cc:503] Master 127.0.205.190:44355 was elected leader, sending a full tablet report...
I20260925 18:12:23.329743 4817 ts_tablet_manager.cc:1428] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260925 18:12:23.332144 4646 heartbeater.cc:503] Master 127.0.205.190:44355 was elected leader, sending a full tablet report...
I20260925 18:12:23.332679 4818 ts_tablet_manager.cc:1428] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260925 18:12:23.420404 4822 raft_consensus.cc:482] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:12:23.420928 4822 raft_consensus.cc:504] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } }
I20260925 18:12:23.422894 4822 leader_election.cc:288] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 298670e239e94e53b0fccb47cff216b8 (127.0.205.131:40455), 63c5ea5ac9074f0cb2459f3bd81016a7 (127.0.205.129:40949)
I20260925 18:12:23.435391 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf0c60abf94c79a10f61aef0136864" candidate_uuid: "78827e238b574577a7ae31cbdb1b9c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "298670e239e94e53b0fccb47cff216b8" is_pre_election: true
I20260925 18:12:23.436280 4770 raft_consensus.cc:2454] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78827e238b574577a7ae31cbdb1b9c66 in term 0.
I20260925 18:12:23.436914 4621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf0c60abf94c79a10f61aef0136864" candidate_uuid: "78827e238b574577a7ae31cbdb1b9c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" is_pre_election: true
I20260925 18:12:23.437469 4621 raft_consensus.cc:2454] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78827e238b574577a7ae31cbdb1b9c66 in term 0.
I20260925 18:12:23.438294 4659 leader_election.cc:302] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63c5ea5ac9074f0cb2459f3bd81016a7, 78827e238b574577a7ae31cbdb1b9c66; no voters:
I20260925 18:12:23.438992 4822 raft_consensus.cc:2781] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 18:12:23.439313 4822 raft_consensus.cc:482] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:12:23.439554 4822 raft_consensus.cc:3037] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:23.443920 4822 raft_consensus.cc:504] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } }
I20260925 18:12:23.445379 4822 leader_election.cc:288] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [CANDIDATE]: Term 1 election: Requested vote from peers 298670e239e94e53b0fccb47cff216b8 (127.0.205.131:40455), 63c5ea5ac9074f0cb2459f3bd81016a7 (127.0.205.129:40949)
I20260925 18:12:23.446151 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf0c60abf94c79a10f61aef0136864" candidate_uuid: "78827e238b574577a7ae31cbdb1b9c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "298670e239e94e53b0fccb47cff216b8"
I20260925 18:12:23.446326 4621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bf0c60abf94c79a10f61aef0136864" candidate_uuid: "78827e238b574577a7ae31cbdb1b9c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7"
I20260925 18:12:23.446590 4770 raft_consensus.cc:3037] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:23.446802 4621 raft_consensus.cc:3037] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:23.451118 4770 raft_consensus.cc:2454] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78827e238b574577a7ae31cbdb1b9c66 in term 1.
I20260925 18:12:23.451251 4621 raft_consensus.cc:2454] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78827e238b574577a7ae31cbdb1b9c66 in term 1.
I20260925 18:12:23.452175 4661 leader_election.cc:302] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 298670e239e94e53b0fccb47cff216b8, 78827e238b574577a7ae31cbdb1b9c66; no voters:
I20260925 18:12:23.452807 4822 raft_consensus.cc:2781] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:23.453635 4822 raft_consensus.cc:686] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 1 LEADER]: Becoming Leader. State: Replica: 78827e238b574577a7ae31cbdb1b9c66, State: Running, Role: LEADER
I20260925 18:12:23.454334 4822 consensus_queue.cc:237] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } }
I20260925 18:12:23.460273 4518 catalog_manager.cc:5495] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78827e238b574577a7ae31cbdb1b9c66 (127.0.205.130). New cstate: current_term: 1 leader_uuid: "78827e238b574577a7ae31cbdb1b9c66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } } }
I20260925 18:12:23.702675 4760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:23.705794 4686 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260925 18:12:23.707222 4611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d45656418a9342d1a2f71f5d804d8444 | 127.0.205.190:44355 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
298670e239e94e53b0fccb47cff216b8 | 127.0.205.131:40455 | HEALTHY | <none> | 0 | 0
63c5ea5ac9074f0cb2459f3bd81016a7 | 127.0.205.129:40949 | HEALTHY | <none> | 0 | 0
78827e238b574577a7ae31cbdb1b9c66 | 127.0.205.130:44467 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet a3bf0c60abf94c79a10f61aef0136864 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
298670e239e94e53b0fccb47cff216b8 (127.0.205.131:40455): RUNNING
78827e238b574577a7ae31cbdb1b9c66 (127.0.205.130:44467): RUNNING [LEADER]
63c5ea5ac9074f0cb2459f3bd81016a7 (127.0.205.129:40949): RUNNING
All reported replicas are:
A = 298670e239e94e53b0fccb47cff216b8
B = 78827e238b574577a7ae31cbdb1b9c66
C = 63c5ea5ac9074f0cb2459f3bd81016a7
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260925 18:12:23.904019 822 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260925 18:12:24.054168 4854 consensus_queue.cc:1035] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:12:24.069197 4854 consensus_queue.cc:1035] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:12:24.271265 4760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:24.272037 4611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:24.272694 4686 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d45656418a9342d1a2f71f5d804d8444 | 127.0.205.190:44355 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
298670e239e94e53b0fccb47cff216b8 | 127.0.205.131:40455 | HEALTHY | <none> | 0 | 0
63c5ea5ac9074f0cb2459f3bd81016a7 | 127.0.205.129:40949 | HEALTHY | <none> | 0 | 0
78827e238b574577a7ae31cbdb1b9c66 | 127.0.205.130:44467 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260925 18:12:24.452559 822 log_verifier.cc:126] Checking tablet a3bf0c60abf94c79a10f61aef0136864
I20260925 18:12:24.509545 822 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a3bf0c60abf94c79a10f61aef0136864
I20260925 18:12:24.510144 822 tablet_replica.cc:330] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: stopping tablet replica
I20260925 18:12:24.510690 822 raft_consensus.cc:2229] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:24.511062 822 raft_consensus.cc:2258] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 18:12:24.626006 4661 consensus_peers.cc:487] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 -> Peer 298670e239e94e53b0fccb47cff216b8 (127.0.205.131:40455): Couldn't send request to peer 298670e239e94e53b0fccb47cff216b8. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260925 18:12:24.627139 4896 raft_consensus.cc:998] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66: Attempting to remove follower 298670e239e94e53b0fccb47cff216b8 from the Raft config. Reason: The tablet replica hosted on peer 298670e239e94e53b0fccb47cff216b8 has failed
I20260925 18:12:24.628782 4896 consensus_queue.cc:237] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } }
I20260925 18:12:24.632586 4621 raft_consensus.cc:1261] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 1 FOLLOWER]: Refusing update from remote peer 78827e238b574577a7ae31cbdb1b9c66: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 18:12:24.633780 4897 consensus_queue.cc:1035] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 18:12:24.639868 4896 raft_consensus.cc:2932] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 298670e239e94e53b0fccb47cff216b8 (127.0.205.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } } }
I20260925 18:12:24.641392 4621 raft_consensus.cc:2932] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 298670e239e94e53b0fccb47cff216b8 (127.0.205.131) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } } }
I20260925 18:12:24.653131 4518 catalog_manager.cc:5495] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 reported cstate change: config changed from index -1 to 2, VOTER 298670e239e94e53b0fccb47cff216b8 (127.0.205.131) evicted. New cstate: current_term: 1 leader_uuid: "78827e238b574577a7ae31cbdb1b9c66" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } } }
I20260925 18:12:24.668380 4760 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bf0c60abf94c79a10f61aef0136864 with delete_type TABLET_DATA_TOMBSTONED (TS 298670e239e94e53b0fccb47cff216b8 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:42226
I20260925 18:12:24.670317 4913 ts_tablet_manager.cc:1905] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 18:12:24.670099 4696 consensus_queue.cc:237] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } } peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } }
W20260925 18:12:24.675612 4661 consensus_peers.cc:487] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 -> Peer 298670e239e94e53b0fccb47cff216b8 (127.0.205.131:40455): Couldn't send request to peer 298670e239e94e53b0fccb47cff216b8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20260925 18:12:24.675967 4621 raft_consensus.cc:1261] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 1 FOLLOWER]: Refusing update from remote peer 78827e238b574577a7ae31cbdb1b9c66: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260925 18:12:24.677596 4896 consensus_queue.cc:1035] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:12:24.685681 4913 ts_tablet_manager.cc:1918] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260925 18:12:24.686256 4913 log.cc:1196] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Deleting WAL directory at /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1790359881141916-822-0/minicluster-data/ts-2-root/wals/a3bf0c60abf94c79a10f61aef0136864
I20260925 18:12:24.686082 4897 raft_consensus.cc:2932] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 298670e239e94e53b0fccb47cff216b8 (127.0.205.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } } peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } }
I20260925 18:12:24.687654 4505 catalog_manager.cc:4841] TS 298670e239e94e53b0fccb47cff216b8 (127.0.205.131:40455): tablet a3bf0c60abf94c79a10f61aef0136864 (table test-workload [id=42f5d6a142674df086c6e6d0079d0193]) successfully deleted
I20260925 18:12:24.687668 4621 raft_consensus.cc:2932] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 298670e239e94e53b0fccb47cff216b8 (127.0.205.131) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } } peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } }
I20260925 18:12:24.694519 4504 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:VOTER RPC for tablet a3bf0c60abf94c79a10f61aef0136864 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20260925 18:12:24.699498 4518 catalog_manager.cc:5495] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 reported cstate change: config changed from index 2 to 3, VOTER 298670e239e94e53b0fccb47cff216b8 (127.0.205.131) added. New cstate: current_term: 1 leader_uuid: "78827e238b574577a7ae31cbdb1b9c66" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } } peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } } }
I20260925 18:12:24.733676 4760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:24.736356 4611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:24.741886 4686 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d45656418a9342d1a2f71f5d804d8444 | 127.0.205.190:44355 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
298670e239e94e53b0fccb47cff216b8 | 127.0.205.131:40455 | HEALTHY | <none> | 0 | 0
63c5ea5ac9074f0cb2459f3bd81016a7 | 127.0.205.129:40949 | HEALTHY | <none> | 0 | 0
78827e238b574577a7ae31cbdb1b9c66 | 127.0.205.130:44467 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet a3bf0c60abf94c79a10f61aef0136864 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
78827e238b574577a7ae31cbdb1b9c66 (127.0.205.130:44467): RUNNING [LEADER]
63c5ea5ac9074f0cb2459f3bd81016a7 (127.0.205.129:40949): RUNNING
All reported replicas are:
A = 78827e238b574577a7ae31cbdb1b9c66
B = 63c5ea5ac9074f0cb2459f3bd81016a7
C = 298670e239e94e53b0fccb47cff216b8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 2
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260925 18:12:24.911134 822 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260925 18:12:25.157706 4928 ts_tablet_manager.cc:927] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Initiating tablet copy from peer 78827e238b574577a7ae31cbdb1b9c66 (127.0.205.130:44467)
I20260925 18:12:25.159291 4928 tablet_copy_client.cc:316] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.205.130:44467
I20260925 18:12:25.170058 4706 tablet_copy_service.cc:140] P 78827e238b574577a7ae31cbdb1b9c66: Received BeginTabletCopySession request for tablet a3bf0c60abf94c79a10f61aef0136864 from peer 298670e239e94e53b0fccb47cff216b8 ({username='slave'} at 127.0.0.1:40382)
I20260925 18:12:25.170604 4706 tablet_copy_service.cc:161] P 78827e238b574577a7ae31cbdb1b9c66: Beginning new tablet copy session on tablet a3bf0c60abf94c79a10f61aef0136864 from peer 298670e239e94e53b0fccb47cff216b8 at {username='slave'} at 127.0.0.1:40382: session id = 298670e239e94e53b0fccb47cff216b8-a3bf0c60abf94c79a10f61aef0136864
I20260925 18:12:25.178220 4706 tablet_copy_source_session.cc:205] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 18:12:25.180612 4928 ts_tablet_manager.cc:1905] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260925 18:12:25.192255 4928 ts_tablet_manager.cc:1918] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20260925 18:12:25.192917 4928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bf0c60abf94c79a10f61aef0136864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:25.199085 4928 tablet_copy_client.cc:791] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: tablet copy: Starting download of 0 data blocks...
I20260925 18:12:25.200646 4928 tablet_copy_client.cc:655] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: tablet copy: Starting download of 1 WAL segments...
I20260925 18:12:25.204089 4928 tablet_copy_client.cc:531] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 18:12:25.211817 4928 tablet_bootstrap.cc:492] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Bootstrap starting.
I20260925 18:12:25.232475 4928 tablet_bootstrap.cc:492] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 18:12:25.233173 4928 tablet_bootstrap.cc:492] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Bootstrap complete.
I20260925 18:12:25.233762 4928 ts_tablet_manager.cc:1397] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.004s
I20260925 18:12:25.235960 4928 raft_consensus.cc:348] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } } peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } }
I20260925 18:12:25.236670 4928 raft_consensus.cc:729] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298670e239e94e53b0fccb47cff216b8, State: Initialized, Role: FOLLOWER
I20260925 18:12:25.237286 4928 consensus_queue.cc:260] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "78827e238b574577a7ae31cbdb1b9c66" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 44467 } } peers { permanent_uuid: "63c5ea5ac9074f0cb2459f3bd81016a7" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 40949 } } peers { permanent_uuid: "298670e239e94e53b0fccb47cff216b8" member_type: VOTER last_known_addr { host: "127.0.205.131" port: 40455 } }
I20260925 18:12:25.239382 4928 ts_tablet_manager.cc:1428] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 18:12:25.241364 4706 tablet_copy_service.cc:342] P 78827e238b574577a7ae31cbdb1b9c66: Request end of tablet copy session 298670e239e94e53b0fccb47cff216b8-a3bf0c60abf94c79a10f61aef0136864 received from {username='slave'} at 127.0.0.1:40382
I20260925 18:12:25.241737 4706 tablet_copy_service.cc:433] P 78827e238b574577a7ae31cbdb1b9c66: ending tablet copy session 298670e239e94e53b0fccb47cff216b8-a3bf0c60abf94c79a10f61aef0136864 on tablet a3bf0c60abf94c79a10f61aef0136864 with peer 298670e239e94e53b0fccb47cff216b8
I20260925 18:12:25.288200 4611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:25.290580 4686 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260925 18:12:25.291117 4760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d45656418a9342d1a2f71f5d804d8444 | 127.0.205.190:44355 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
298670e239e94e53b0fccb47cff216b8 | 127.0.205.131:40455 | HEALTHY | <none> | 0 | 0
63c5ea5ac9074f0cb2459f3bd81016a7 | 127.0.205.129:40949 | HEALTHY | <none> | 0 | 0
78827e238b574577a7ae31cbdb1b9c66 | 127.0.205.130:44467 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet a3bf0c60abf94c79a10f61aef0136864 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
78827e238b574577a7ae31cbdb1b9c66 (127.0.205.130:44467): RUNNING [LEADER]
63c5ea5ac9074f0cb2459f3bd81016a7 (127.0.205.129:40949): RUNNING
298670e239e94e53b0fccb47cff216b8 (127.0.205.131:40455): RUNNING
All reported replicas are:
A = 78827e238b574577a7ae31cbdb1b9c66
B = 63c5ea5ac9074f0cb2459f3bd81016a7
C = 298670e239e94e53b0fccb47cff216b8
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | 3 | Yes
B | A* B C | 1 | 3 | Yes
C | A B C | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260925 18:12:25.482443 822 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260925 18:12:25.561782 4770 raft_consensus.cc:1203] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20260925 18:12:25.908716 4760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:25.920332 4611 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:25.921634 4686 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
d45656418a9342d1a2f71f5d804d8444 | 127.0.205.190:44355 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
298670e239e94e53b0fccb47cff216b8 | 127.0.205.131:40455 | HEALTHY | <none> | 0 | 0
63c5ea5ac9074f0cb2459f3bd81016a7 | 127.0.205.129:40949 | HEALTHY | <none> | 0 | 0
78827e238b574577a7ae31cbdb1b9c66 | 127.0.205.130:44467 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260925 18:12:26.089494 822 log_verifier.cc:126] Checking tablet a3bf0c60abf94c79a10f61aef0136864
I20260925 18:12:26.147310 822 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a3bf0c60abf94c79a10f61aef0136864
I20260925 18:12:26.151111 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:12:26.167933 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:26.168542 822 tablet_replica.cc:330] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7: stopping tablet replica
I20260925 18:12:26.169080 822 raft_consensus.cc:2229] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:26.169543 822 raft_consensus.cc:2258] T a3bf0c60abf94c79a10f61aef0136864 P 63c5ea5ac9074f0cb2459f3bd81016a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:26.187961 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:12:26.196506 822 tablet_server.cc:178] TabletServer@127.0.205.130:0 shutting down...
I20260925 18:12:26.211231 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:26.211808 822 tablet_replica.cc:330] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66: stopping tablet replica
I20260925 18:12:26.212343 822 raft_consensus.cc:2229] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:26.213106 822 raft_consensus.cc:2258] T a3bf0c60abf94c79a10f61aef0136864 P 78827e238b574577a7ae31cbdb1b9c66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:26.231918 822 tablet_server.cc:194] TabletServer@127.0.205.130:0 shutdown complete.
I20260925 18:12:26.240198 822 tablet_server.cc:178] TabletServer@127.0.205.131:0 shutting down...
I20260925 18:12:26.256237 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:26.256820 822 tablet_replica.cc:330] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8: stopping tablet replica
I20260925 18:12:26.257349 822 raft_consensus.cc:2229] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:26.257704 822 raft_consensus.cc:2258] T a3bf0c60abf94c79a10f61aef0136864 P 298670e239e94e53b0fccb47cff216b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:26.275568 822 tablet_server.cc:194] TabletServer@127.0.205.131:0 shutdown complete.
I20260925 18:12:26.283772 822 master.cc:537] Master@127.0.205.190:44355 shutting down...
I20260925 18:12:26.296598 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:26.297224 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:26.297533 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P d45656418a9342d1a2f71f5d804d8444: stopping tablet replica
I20260925 18:12:26.314208 822 master.cc:552] Master@127.0.205.190:44355 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (3636 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20260925 18:12:26.338236 822 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.205.190:34289
I20260925 18:12:26.339171 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:26.343972 4981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:26.344898 4982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:26.346158 4984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:26.346459 822 server_base.cc:1034] running on GCE node
I20260925 18:12:26.347143 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:26.347302 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:26.347412 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359946347404 us; error 0 us; skew 500 ppm
I20260925 18:12:26.347832 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:26.350052 822 webserver.cc:460] Webserver started at http://127.0.205.190:38427/ using document root <none> and password file <none>
I20260925 18:12:26.350484 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:26.350631 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:26.350818 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:26.351799 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/master-0-root/instance:
uuid: "b22ef9812bc047579902b4223356005f"
format_stamp: "Formatted at 2026-09-25 18:12:26 on dist-test-slave-57m3"
I20260925 18:12:26.355886 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 18:12:26.359117 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.359800 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:12:26.360054 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/master-0-root
uuid: "b22ef9812bc047579902b4223356005f"
format_stamp: "Formatted at 2026-09-25 18:12:26 on dist-test-slave-57m3"
I20260925 18:12:26.360312 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:26.376133 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:26.377097 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:26.408810 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.190:34289
I20260925 18:12:26.408921 5040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.190:34289 every 8 connection(s)
I20260925 18:12:26.412235 5041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:26.421522 5041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f: Bootstrap starting.
I20260925 18:12:26.425346 5041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:26.428975 5041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f: No bootstrap required, opened a new log
I20260925 18:12:26.430708 5041 raft_consensus.cc:348] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22ef9812bc047579902b4223356005f" member_type: VOTER }
I20260925 18:12:26.431062 5041 raft_consensus.cc:374] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:26.431272 5041 raft_consensus.cc:729] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22ef9812bc047579902b4223356005f, State: Initialized, Role: FOLLOWER
I20260925 18:12:26.431829 5041 consensus_queue.cc:260] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22ef9812bc047579902b4223356005f" member_type: VOTER }
I20260925 18:12:26.432241 5041 raft_consensus.cc:388] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 18:12:26.432435 5041 raft_consensus.cc:482] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 18:12:26.432675 5041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:26.437062 5041 raft_consensus.cc:504] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22ef9812bc047579902b4223356005f" member_type: VOTER }
I20260925 18:12:26.437556 5041 leader_election.cc:302] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b22ef9812bc047579902b4223356005f; no voters:
I20260925 18:12:26.438567 5041 leader_election.cc:288] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 18:12:26.438843 5044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:26.440026 5044 raft_consensus.cc:686] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 1 LEADER]: Becoming Leader. State: Replica: b22ef9812bc047579902b4223356005f, State: Running, Role: LEADER
I20260925 18:12:26.440622 5044 consensus_queue.cc:237] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22ef9812bc047579902b4223356005f" member_type: VOTER }
I20260925 18:12:26.441214 5041 sys_catalog.cc:564] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 18:12:26.443549 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b22ef9812bc047579902b4223356005f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22ef9812bc047579902b4223356005f" member_type: VOTER } }
I20260925 18:12:26.443661 5046 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b22ef9812bc047579902b4223356005f. Latest consensus state: current_term: 1 leader_uuid: "b22ef9812bc047579902b4223356005f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22ef9812bc047579902b4223356005f" member_type: VOTER } }
I20260925 18:12:26.444164 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:26.444186 5046 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [sys.catalog]: This master's current role is: LEADER
I20260925 18:12:26.446687 5050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 18:12:26.453218 5050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 18:12:26.454418 822 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260925 18:12:26.460891 5050 catalog_manager.cc:1347] Generated new cluster ID: d153d4e217ca40f09257c773b983488e
I20260925 18:12:26.461122 5050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 18:12:26.487277 5050 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 18:12:26.488956 5050 catalog_manager.cc:1504] Loading token signing keys...
I20260925 18:12:26.500362 5050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f: Generated new TSK 0
I20260925 18:12:26.501006 5050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 18:12:26.520539 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:26.526007 5062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:26.526748 5063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:26.528199 5065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:26.528653 822 server_base.cc:1034] running on GCE node
I20260925 18:12:26.529423 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:26.529585 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:26.529696 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359946529688 us; error 0 us; skew 500 ppm
I20260925 18:12:26.530112 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:26.532171 822 webserver.cc:460] Webserver started at http://127.0.205.129:42717/ using document root <none> and password file <none>
I20260925 18:12:26.532568 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:26.532727 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:26.532989 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:26.533953 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-0-root/instance:
uuid: "0542a42032584d74ae62283be4cae3c5"
format_stamp: "Formatted at 2026-09-25 18:12:26 on dist-test-slave-57m3"
I20260925 18:12:26.538009 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 18:12:26.540858 5070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.541596 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:12:26.541839 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-0-root
uuid: "0542a42032584d74ae62283be4cae3c5"
format_stamp: "Formatted at 2026-09-25 18:12:26 on dist-test-slave-57m3"
I20260925 18:12:26.542078 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:26.561676 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:26.562685 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:26.563982 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:12:26.566047 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:26.566211 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.566438 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:26.566569 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.601428 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.129:41355
I20260925 18:12:26.601503 5132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.129:41355 every 8 connection(s)
I20260925 18:12:26.605918 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:26.613723 5137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:26.614800 5138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:26.616499 5140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:26.617477 822 server_base.cc:1034] running on GCE node
I20260925 18:12:26.618196 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:26.618376 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:26.618536 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359946618524 us; error 0 us; skew 500 ppm
I20260925 18:12:26.619032 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:26.620425 5133 heartbeater.cc:347] Connected to a master server at 127.0.205.190:34289
I20260925 18:12:26.620807 5133 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:26.621671 5133 heartbeater.cc:511] Master 127.0.205.190:34289 requested a full tablet report, sending...
I20260925 18:12:26.621726 822 webserver.cc:460] Webserver started at http://127.0.205.130:43131/ using document root <none> and password file <none>
I20260925 18:12:26.622393 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:26.622607 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:26.622870 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:26.623721 5006 ts_manager.cc:194] Registered new tserver with Master: 0542a42032584d74ae62283be4cae3c5 (127.0.205.129:41355)
I20260925 18:12:26.623960 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-1-root/instance:
uuid: "dada11d42d9148ca80d4436a175e5b10"
format_stamp: "Formatted at 2026-09-25 18:12:26 on dist-test-slave-57m3"
I20260925 18:12:26.625612 5006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32888
I20260925 18:12:26.628551 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 18:12:26.631629 5145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.632359 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:12:26.632613 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-1-root
uuid: "dada11d42d9148ca80d4436a175e5b10"
format_stamp: "Formatted at 2026-09-25 18:12:26 on dist-test-slave-57m3"
I20260925 18:12:26.632902 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:26.651433 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:26.652432 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:26.653767 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:12:26.655850 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:26.656018 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.656239 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:26.656371 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.690760 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.130:35655
I20260925 18:12:26.690845 5207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.130:35655 every 8 connection(s)
I20260925 18:12:26.695078 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:26.701653 5211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:26.703651 5212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:26.706195 5214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:26.706387 822 server_base.cc:1034] running on GCE node
I20260925 18:12:26.707525 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:26.707788 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:26.707940 5208 heartbeater.cc:347] Connected to a master server at 127.0.205.190:34289
I20260925 18:12:26.707975 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359946707959 us; error 0 us; skew 500 ppm
I20260925 18:12:26.708350 5208 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:26.708716 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:26.708976 5208 heartbeater.cc:511] Master 127.0.205.190:34289 requested a full tablet report, sending...
I20260925 18:12:26.710842 5006 ts_manager.cc:194] Registered new tserver with Master: dada11d42d9148ca80d4436a175e5b10 (127.0.205.130:35655)
I20260925 18:12:26.711251 822 webserver.cc:460] Webserver started at http://127.0.205.131:41655/ using document root <none> and password file <none>
I20260925 18:12:26.711721 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:26.711889 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:26.712116 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:26.712540 5006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32900
I20260925 18:12:26.713222 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-2-root/instance:
uuid: "d4f2c6961f124baebbeb52958b96c48f"
format_stamp: "Formatted at 2026-09-25 18:12:26 on dist-test-slave-57m3"
I20260925 18:12:26.717432 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 18:12:26.720611 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.721354 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 18:12:26.721635 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-2-root
uuid: "d4f2c6961f124baebbeb52958b96c48f"
format_stamp: "Formatted at 2026-09-25 18:12:26 on dist-test-slave-57m3"
I20260925 18:12:26.721889 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:26.746703 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:26.747738 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:26.749147 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:12:26.751421 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:26.751611 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.751861 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:26.752008 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.786747 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.131:41141
I20260925 18:12:26.786834 5281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.131:41141 every 8 connection(s)
I20260925 18:12:26.791216 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 18:12:26.797618 5285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:26.799220 5286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 18:12:26.799958 5288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 18:12:26.801255 822 server_base.cc:1034] running on GCE node
I20260925 18:12:26.802368 822 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260925 18:12:26.802564 822 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260925 18:12:26.802757 822 hybrid_clock.cc:648] HybridClock initialized: now 1790359946802740 us; error 0 us; skew 500 ppm
I20260925 18:12:26.803416 822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 18:12:26.804909 5282 heartbeater.cc:347] Connected to a master server at 127.0.205.190:34289
I20260925 18:12:26.805251 5282 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:26.805842 5282 heartbeater.cc:511] Master 127.0.205.190:34289 requested a full tablet report, sending...
I20260925 18:12:26.805991 822 webserver.cc:460] Webserver started at http://127.0.205.132:39265/ using document root <none> and password file <none>
I20260925 18:12:26.806492 822 fs_manager.cc:359] Metadata directory not provided
I20260925 18:12:26.806650 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 18:12:26.806892 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 18:12:26.807526 5006 ts_manager.cc:194] Registered new tserver with Master: d4f2c6961f124baebbeb52958b96c48f (127.0.205.131:41141)
I20260925 18:12:26.807971 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-3-root/instance:
uuid: "f9dc8f9a62b742c0b03e47db27765ab8"
format_stamp: "Formatted at 2026-09-25 18:12:26 on dist-test-slave-57m3"
I20260925 18:12:26.809055 5006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32914
I20260925 18:12:26.812201 822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 18:12:26.815338 5293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.816015 822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 18:12:26.816309 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-3-root
uuid: "f9dc8f9a62b742c0b03e47db27765ab8"
format_stamp: "Formatted at 2026-09-25 18:12:26 on dist-test-slave-57m3"
I20260925 18:12:26.816560 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 18:12:26.838153 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 18:12:26.839093 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 18:12:26.840369 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 18:12:26.842545 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 18:12:26.842724 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.842931 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 18:12:26.843060 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 18:12:26.878381 822 rpc_server.cc:307] RPC server started. Bound to: 127.0.205.132:40471
I20260925 18:12:26.878477 5355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.205.132:40471 every 8 connection(s)
I20260925 18:12:26.892482 5356 heartbeater.cc:347] Connected to a master server at 127.0.205.190:34289
I20260925 18:12:26.892869 5356 heartbeater.cc:464] Registering TS with master...
I20260925 18:12:26.893478 5356 heartbeater.cc:511] Master 127.0.205.190:34289 requested a full tablet report, sending...
I20260925 18:12:26.895256 5006 ts_manager.cc:194] Registered new tserver with Master: f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471)
I20260925 18:12:26.896205 822 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.01480128s
I20260925 18:12:26.897112 5006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32924
I20260925 18:12:26.917485 822 test_util.cc:268] Using random seed: -1465387432
I20260925 18:12:26.938652 5006 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32954:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 18:12:26.982369 5173 tablet_service.cc:1467] Processing CreateTablet for tablet ba3f57e7cb32450db190b7448c5175ab (DEFAULT_TABLE table=test-workload [id=6e004258317244a0b148878cda59aa91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:12:26.983613 5173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3f57e7cb32450db190b7448c5175ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:26.987964 5321 tablet_service.cc:1467] Processing CreateTablet for tablet ba3f57e7cb32450db190b7448c5175ab (DEFAULT_TABLE table=test-workload [id=6e004258317244a0b148878cda59aa91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:12:26.987952 5098 tablet_service.cc:1467] Processing CreateTablet for tablet ba3f57e7cb32450db190b7448c5175ab (DEFAULT_TABLE table=test-workload [id=6e004258317244a0b148878cda59aa91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 18:12:26.989178 5321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3f57e7cb32450db190b7448c5175ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:26.989398 5098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3f57e7cb32450db190b7448c5175ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 18:12:26.999682 5376 tablet_bootstrap.cc:492] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10: Bootstrap starting.
I20260925 18:12:27.001744 5377 tablet_bootstrap.cc:492] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8: Bootstrap starting.
I20260925 18:12:27.006093 5376 tablet_bootstrap.cc:654] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:27.007637 5377 tablet_bootstrap.cc:654] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:27.009846 5376 tablet_bootstrap.cc:492] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10: No bootstrap required, opened a new log
I20260925 18:12:27.010238 5376 ts_tablet_manager.cc:1397] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 18:12:27.012032 5377 tablet_bootstrap.cc:492] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8: No bootstrap required, opened a new log
I20260925 18:12:27.012380 5377 ts_tablet_manager.cc:1397] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260925 18:12:27.012818 5378 tablet_bootstrap.cc:492] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5: Bootstrap starting.
I20260925 18:12:27.014348 5377 raft_consensus.cc:348] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:27.014868 5377 raft_consensus.cc:374] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:27.015093 5377 raft_consensus.cc:729] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dc8f9a62b742c0b03e47db27765ab8, State: Initialized, Role: FOLLOWER
I20260925 18:12:27.012324 5376 raft_consensus.cc:348] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:27.015902 5377 consensus_queue.cc:260] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:27.015918 5376 raft_consensus.cc:374] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:27.017485 5376 raft_consensus.cc:729] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dada11d42d9148ca80d4436a175e5b10, State: Initialized, Role: FOLLOWER
I20260925 18:12:27.018529 5378 tablet_bootstrap.cc:654] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5: Neither blocks nor log segments found. Creating new log.
I20260925 18:12:27.018947 5356 heartbeater.cc:503] Master 127.0.205.190:34289 was elected leader, sending a full tablet report...
I20260925 18:12:27.019181 5377 ts_tablet_manager.cc:1428] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 18:12:27.018322 5376 consensus_queue.cc:260] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:27.021880 5208 heartbeater.cc:503] Master 127.0.205.190:34289 was elected leader, sending a full tablet report...
I20260925 18:12:27.024258 5376 ts_tablet_manager.cc:1428] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10: Time spent starting tablet: real 0.014s user 0.006s sys 0.002s
I20260925 18:12:27.027456 5378 tablet_bootstrap.cc:492] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5: No bootstrap required, opened a new log
I20260925 18:12:27.027878 5378 ts_tablet_manager.cc:1397] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260925 18:12:27.030274 5378 raft_consensus.cc:348] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:27.030939 5378 raft_consensus.cc:374] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 18:12:27.031230 5378 raft_consensus.cc:729] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0542a42032584d74ae62283be4cae3c5, State: Initialized, Role: FOLLOWER
I20260925 18:12:27.031883 5378 consensus_queue.cc:260] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:27.033882 5133 heartbeater.cc:503] Master 127.0.205.190:34289 was elected leader, sending a full tablet report...
I20260925 18:12:27.034132 5378 ts_tablet_manager.cc:1428] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 18:12:27.061498 5381 raft_consensus.cc:482] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 18:12:27.061950 5381 raft_consensus.cc:504] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:27.063880 5381 leader_election.cc:288] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dada11d42d9148ca80d4436a175e5b10 (127.0.205.130:35655), 0542a42032584d74ae62283be4cae3c5 (127.0.205.129:41355)
I20260925 18:12:27.076222 5183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3f57e7cb32450db190b7448c5175ab" candidate_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dada11d42d9148ca80d4436a175e5b10" is_pre_election: true
I20260925 18:12:27.076985 5183 raft_consensus.cc:2454] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9dc8f9a62b742c0b03e47db27765ab8 in term 0.
I20260925 18:12:27.077685 5108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3f57e7cb32450db190b7448c5175ab" candidate_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0542a42032584d74ae62283be4cae3c5" is_pre_election: true
I20260925 18:12:27.078084 5294 leader_election.cc:302] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dada11d42d9148ca80d4436a175e5b10, f9dc8f9a62b742c0b03e47db27765ab8; no voters:
I20260925 18:12:27.078397 5108 raft_consensus.cc:2454] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9dc8f9a62b742c0b03e47db27765ab8 in term 0.
I20260925 18:12:27.078878 5381 raft_consensus.cc:2781] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 18:12:27.079186 5381 raft_consensus.cc:482] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 18:12:27.079443 5381 raft_consensus.cc:3037] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:27.083707 5381 raft_consensus.cc:504] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:27.085136 5381 leader_election.cc:288] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [CANDIDATE]: Term 1 election: Requested vote from peers dada11d42d9148ca80d4436a175e5b10 (127.0.205.130:35655), 0542a42032584d74ae62283be4cae3c5 (127.0.205.129:41355)
I20260925 18:12:27.085894 5183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3f57e7cb32450db190b7448c5175ab" candidate_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dada11d42d9148ca80d4436a175e5b10"
I20260925 18:12:27.085989 5108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3f57e7cb32450db190b7448c5175ab" candidate_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0542a42032584d74ae62283be4cae3c5"
I20260925 18:12:27.086364 5183 raft_consensus.cc:3037] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:27.086470 5108 raft_consensus.cc:3037] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 18:12:27.090786 5183 raft_consensus.cc:2454] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9dc8f9a62b742c0b03e47db27765ab8 in term 1.
I20260925 18:12:27.090787 5108 raft_consensus.cc:2454] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9dc8f9a62b742c0b03e47db27765ab8 in term 1.
I20260925 18:12:27.091825 5294 leader_election.cc:302] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dada11d42d9148ca80d4436a175e5b10, f9dc8f9a62b742c0b03e47db27765ab8; no voters:
I20260925 18:12:27.092490 5381 raft_consensus.cc:2781] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 18:12:27.093609 5381 raft_consensus.cc:686] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 1 LEADER]: Becoming Leader. State: Replica: f9dc8f9a62b742c0b03e47db27765ab8, State: Running, Role: LEADER
I20260925 18:12:27.094399 5381 consensus_queue.cc:237] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:27.100063 5005 catalog_manager.cc:5495] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132). New cstate: current_term: 1 leader_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } health_report { overall_health: UNKNOWN } } }
I20260925 18:12:27.343416 5098 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:27.360327 5173 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:27.360222 5247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:27.369007 5321 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b22ef9812bc047579902b4223356005f | 127.0.205.190:34289 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0542a42032584d74ae62283be4cae3c5 | 127.0.205.129:41355 | HEALTHY | <none> | 0 | 0
d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | HEALTHY | <none> | 0 | 0
dada11d42d9148ca80d4436a175e5b10 | 127.0.205.130:35655 | HEALTHY | <none> | 0 | 0
f9dc8f9a62b742c0b03e47db27765ab8 | 127.0.205.132:40471 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet ba3f57e7cb32450db190b7448c5175ab of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
dada11d42d9148ca80d4436a175e5b10 (127.0.205.130:35655): RUNNING
f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471): RUNNING [LEADER]
0542a42032584d74ae62283be4cae3c5 (127.0.205.129:41355): RUNNING
All reported replicas are:
A = dada11d42d9148ca80d4436a175e5b10
B = f9dc8f9a62b742c0b03e47db27765ab8
C = 0542a42032584d74ae62283be4cae3c5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260925 18:12:27.595089 5381 consensus_queue.cc:1035] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:12:27.600148 822 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260925 18:12:27.601636 5381 consensus_queue.cc:1035] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 18:12:27.811607 5282 heartbeater.cc:503] Master 127.0.205.190:34289 was elected leader, sending a full tablet report...
I20260925 18:12:27.961164 5098 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:27.971392 5173 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:27.972364 5321 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260925 18:12:27.977115 5247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b22ef9812bc047579902b4223356005f | 127.0.205.190:34289 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0542a42032584d74ae62283be4cae3c5 | 127.0.205.129:41355 | HEALTHY | <none> | 0 | 0
d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | HEALTHY | <none> | 0 | 0
dada11d42d9148ca80d4436a175e5b10 | 127.0.205.130:35655 | HEALTHY | <none> | 0 | 0
f9dc8f9a62b742c0b03e47db27765ab8 | 127.0.205.132:40471 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260925 18:12:28.221045 822 log_verifier.cc:126] Checking tablet ba3f57e7cb32450db190b7448c5175ab
I20260925 18:12:28.278225 822 log_verifier.cc:177] Verified matching terms for 1 ops in tablet ba3f57e7cb32450db190b7448c5175ab
I20260925 18:12:28.282382 822 tablet_replica.cc:330] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8: stopping tablet replica
I20260925 18:12:28.282878 822 raft_consensus.cc:2229] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:28.283568 822 raft_consensus.cc:2258] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:28.487147 5098 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:28.496289 5247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:28.504884 5173 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:28.505815 5321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b22ef9812bc047579902b4223356005f | 127.0.205.190:34289 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0542a42032584d74ae62283be4cae3c5 | 127.0.205.129:41355 | HEALTHY | <none> | 0 | 0
d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | HEALTHY | <none> | 0 | 0
dada11d42d9148ca80d4436a175e5b10 | 127.0.205.130:35655 | HEALTHY | <none> | 0 | 0
f9dc8f9a62b742c0b03e47db27765ab8 | 127.0.205.132:40471 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet ba3f57e7cb32450db190b7448c5175ab of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
dada11d42d9148ca80d4436a175e5b10 (127.0.205.130:35655): RUNNING
f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
0542a42032584d74ae62283be4cae3c5 (127.0.205.129:41355): RUNNING
All reported replicas are:
A = dada11d42d9148ca80d4436a175e5b10
B = f9dc8f9a62b742c0b03e47db27765ab8
C = 0542a42032584d74ae62283be4cae3c5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260925 18:12:28.744581 822 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260925 18:12:29.114884 5098 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:29.115119 5173 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:29.115392 5247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:29.121336 5321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b22ef9812bc047579902b4223356005f | 127.0.205.190:34289 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0542a42032584d74ae62283be4cae3c5 | 127.0.205.129:41355 | HEALTHY | <none> | 0 | 0
d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | HEALTHY | <none> | 0 | 0
dada11d42d9148ca80d4436a175e5b10 | 127.0.205.130:35655 | HEALTHY | <none> | 0 | 0
f9dc8f9a62b742c0b03e47db27765ab8 | 127.0.205.132:40471 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet ba3f57e7cb32450db190b7448c5175ab of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
dada11d42d9148ca80d4436a175e5b10 (127.0.205.130:35655): RUNNING
f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
0542a42032584d74ae62283be4cae3c5 (127.0.205.129:41355): RUNNING
All reported replicas are:
A = dada11d42d9148ca80d4436a175e5b10
B = f9dc8f9a62b742c0b03e47db27765ab8
C = 0542a42032584d74ae62283be4cae3c5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | -1 | Yes
B | [config not available] | | |
C | A B* C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260925 18:12:29.351781 822 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260925 18:12:29.583503 5519 raft_consensus.cc:482] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9dc8f9a62b742c0b03e47db27765ab8)
I20260925 18:12:29.583974 5519 raft_consensus.cc:504] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:29.586135 5519 leader_election.cc:288] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dada11d42d9148ca80d4436a175e5b10 (127.0.205.130:35655), f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471)
I20260925 18:12:29.603086 5183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3f57e7cb32450db190b7448c5175ab" candidate_uuid: "0542a42032584d74ae62283be4cae3c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dada11d42d9148ca80d4436a175e5b10" is_pre_election: true
I20260925 18:12:29.605551 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3f57e7cb32450db190b7448c5175ab" candidate_uuid: "0542a42032584d74ae62283be4cae3c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" is_pre_election: true
W20260925 18:12:29.606737 5071 leader_election.cc:341] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471): Illegal state: cannot vote while shut down
I20260925 18:12:29.607096 5071 leader_election.cc:302] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0542a42032584d74ae62283be4cae3c5; no voters: dada11d42d9148ca80d4436a175e5b10, f9dc8f9a62b742c0b03e47db27765ab8
I20260925 18:12:29.607813 5519 raft_consensus.cc:2726] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 18:12:29.705524 5524 raft_consensus.cc:482] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9dc8f9a62b742c0b03e47db27765ab8)
I20260925 18:12:29.706019 5524 raft_consensus.cc:504] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:29.707860 5524 leader_election.cc:288] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471), 0542a42032584d74ae62283be4cae3c5 (127.0.205.129:41355)
I20260925 18:12:29.725324 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3f57e7cb32450db190b7448c5175ab" candidate_uuid: "dada11d42d9148ca80d4436a175e5b10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" is_pre_election: true
W20260925 18:12:29.726930 5146 leader_election.cc:341] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471): Illegal state: cannot vote while shut down
I20260925 18:12:29.728242 5108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3f57e7cb32450db190b7448c5175ab" candidate_uuid: "dada11d42d9148ca80d4436a175e5b10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0542a42032584d74ae62283be4cae3c5" is_pre_election: true
I20260925 18:12:29.728950 5108 raft_consensus.cc:2454] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dada11d42d9148ca80d4436a175e5b10 in term 1.
I20260925 18:12:29.730095 5147 leader_election.cc:302] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0542a42032584d74ae62283be4cae3c5, dada11d42d9148ca80d4436a175e5b10; no voters: f9dc8f9a62b742c0b03e47db27765ab8
I20260925 18:12:29.730849 5524 raft_consensus.cc:2781] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 18:12:29.731364 5524 raft_consensus.cc:482] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9dc8f9a62b742c0b03e47db27765ab8)
I20260925 18:12:29.731719 5524 raft_consensus.cc:3037] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 1 FOLLOWER]: Advancing to term 2
I20260925 18:12:29.737685 5524 raft_consensus.cc:504] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:29.739122 5524 leader_election.cc:288] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [CANDIDATE]: Term 2 election: Requested vote from peers f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471), 0542a42032584d74ae62283be4cae3c5 (127.0.205.129:41355)
I20260925 18:12:29.739980 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3f57e7cb32450db190b7448c5175ab" candidate_uuid: "dada11d42d9148ca80d4436a175e5b10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9dc8f9a62b742c0b03e47db27765ab8"
I20260925 18:12:29.740180 5108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3f57e7cb32450db190b7448c5175ab" candidate_uuid: "dada11d42d9148ca80d4436a175e5b10" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0542a42032584d74ae62283be4cae3c5"
I20260925 18:12:29.740819 5108 raft_consensus.cc:3037] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 1 FOLLOWER]: Advancing to term 2
W20260925 18:12:29.741199 5146 leader_election.cc:341] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471): Illegal state: cannot vote while shut down
I20260925 18:12:29.747730 5108 raft_consensus.cc:2454] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dada11d42d9148ca80d4436a175e5b10 in term 2.
I20260925 18:12:29.748694 5147 leader_election.cc:302] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0542a42032584d74ae62283be4cae3c5, dada11d42d9148ca80d4436a175e5b10; no voters: f9dc8f9a62b742c0b03e47db27765ab8
I20260925 18:12:29.749562 5524 raft_consensus.cc:2781] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 18:12:29.750600 5524 raft_consensus.cc:686] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 2 LEADER]: Becoming Leader. State: Replica: dada11d42d9148ca80d4436a175e5b10, State: Running, Role: LEADER
I20260925 18:12:29.751426 5524 consensus_queue.cc:237] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:29.758236 5003 catalog_manager.cc:5495] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 reported cstate change: term changed from 1 to 2, leader changed from f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132) to dada11d42d9148ca80d4436a175e5b10 (127.0.205.130). New cstate: current_term: 2 leader_uuid: "dada11d42d9148ca80d4436a175e5b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9dc8f9a62b742c0b03e47db27765ab8" member_type: VOTER last_known_addr { host: "127.0.205.132" port: 40471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } health_report { overall_health: UNKNOWN } } }
I20260925 18:12:29.812103 5098 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:29.829160 5173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260925 18:12:29.836618 5247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:29.850185 5321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b22ef9812bc047579902b4223356005f | 127.0.205.190:34289 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0542a42032584d74ae62283be4cae3c5 | 127.0.205.129:41355 | HEALTHY | <none> | 0 | 0
d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | HEALTHY | <none> | 0 | 0
dada11d42d9148ca80d4436a175e5b10 | 127.0.205.130:35655 | HEALTHY | <none> | 1 | 0
f9dc8f9a62b742c0b03e47db27765ab8 | 127.0.205.132:40471 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet ba3f57e7cb32450db190b7448c5175ab of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
dada11d42d9148ca80d4436a175e5b10 (127.0.205.130:35655): RUNNING [LEADER]
f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
0542a42032584d74ae62283be4cae3c5 (127.0.205.129:41355): RUNNING
All reported replicas are:
A = dada11d42d9148ca80d4436a175e5b10
B = f9dc8f9a62b742c0b03e47db27765ab8
C = 0542a42032584d74ae62283be4cae3c5
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | [config not available] | | |
C | A B C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20260925 18:12:30.055492 822 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260925 18:12:30.150122 5146 consensus_queue.cc:579] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [LEADER]: The tablet replica hosted on peer f9dc8f9a62b742c0b03e47db27765ab8 has failed
W20260925 18:12:30.150547 5146 consensus_peers.cc:487] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 -> Peer f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471): Couldn't send request to peer f9dc8f9a62b742c0b03e47db27765ab8. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20260925 18:12:30.209561 5108 raft_consensus.cc:1261] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 2 FOLLOWER]: Refusing update from remote peer dada11d42d9148ca80d4436a175e5b10: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260925 18:12:30.210842 5524 consensus_queue.cc:1035] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260925 18:12:30.226908 5183 consensus_queue.cc:237] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } }
I20260925 18:12:30.230193 5108 raft_consensus.cc:1261] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 2 FOLLOWER]: Refusing update from remote peer dada11d42d9148ca80d4436a175e5b10: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 18:12:30.231181 5524 consensus_queue.cc:1035] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 18:12:30.235831 5538 raft_consensus.cc:2932] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } } }
I20260925 18:12:30.237330 5108 raft_consensus.cc:2932] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } } }
I20260925 18:12:30.242292 4990 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ba3f57e7cb32450db190b7448c5175ab with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260925 18:12:30.245688 5004 catalog_manager.cc:5495] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 reported cstate change: config changed from index -1 to 3, VOTER f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132) evicted. New cstate: current_term: 2 leader_uuid: "dada11d42d9148ca80d4436a175e5b10" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } health_report { overall_health: HEALTHY } } }
I20260925 18:12:30.254096 5321 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3f57e7cb32450db190b7448c5175ab with delete_type TABLET_DATA_TOMBSTONED (TS f9dc8f9a62b742c0b03e47db27765ab8 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:60794
I20260925 18:12:30.255879 5557 ts_tablet_manager.cc:1905] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260925 18:12:30.257395 5183 consensus_queue.cc:237] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } } peers { permanent_uuid: "d4f2c6961f124baebbeb52958b96c48f" member_type: NON_VOTER last_known_addr { host: "127.0.205.131" port: 41141 } attrs { promote: true } }
I20260925 18:12:30.263648 5108 raft_consensus.cc:1261] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 2 FOLLOWER]: Refusing update from remote peer dada11d42d9148ca80d4436a175e5b10: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260925 18:12:30.265197 5538 consensus_queue.cc:1035] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260925 18:12:30.269060 5557 ts_tablet_manager.cc:1918] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20260925 18:12:30.269470 5557 log.cc:1196] T ba3f57e7cb32450db190b7448c5175ab P f9dc8f9a62b742c0b03e47db27765ab8: Deleting WAL directory at /tmp/dist-test-taskwNiUry/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1790359881141916-822-0/minicluster-data/ts-3-root/wals/ba3f57e7cb32450db190b7448c5175ab
I20260925 18:12:30.271266 4990 catalog_manager.cc:4841] TS f9dc8f9a62b742c0b03e47db27765ab8 (127.0.205.132:40471): tablet ba3f57e7cb32450db190b7448c5175ab (table test-workload [id=6e004258317244a0b148878cda59aa91]) successfully deleted
I20260925 18:12:30.270769 5524 raft_consensus.cc:2932] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER d4f2c6961f124baebbeb52958b96c48f (127.0.205.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } } peers { permanent_uuid: "d4f2c6961f124baebbeb52958b96c48f" member_type: NON_VOTER last_known_addr { host: "127.0.205.131" port: 41141 } attrs { promote: true } } }
I20260925 18:12:30.272321 5108 raft_consensus.cc:2932] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, NON_VOTER d4f2c6961f124baebbeb52958b96c48f (127.0.205.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } } peers { permanent_uuid: "d4f2c6961f124baebbeb52958b96c48f" member_type: NON_VOTER last_known_addr { host: "127.0.205.131" port: 41141 } attrs { promote: true } } }
W20260925 18:12:30.275079 5148 consensus_peers.cc:487] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 -> Peer d4f2c6961f124baebbeb52958b96c48f (127.0.205.131:41141): Couldn't send request to peer d4f2c6961f124baebbeb52958b96c48f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba3f57e7cb32450db190b7448c5175ab. This is attempt 1: this message will repeat every 5th retry.
I20260925 18:12:30.279805 4990 catalog_manager.cc:5008] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ba3f57e7cb32450db190b7448c5175ab with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20260925 18:12:30.281386 5004 catalog_manager.cc:5495] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 reported cstate change: config changed from index 3 to 4, NON_VOTER d4f2c6961f124baebbeb52958b96c48f (127.0.205.131) added. New cstate: current_term: 2 leader_uuid: "dada11d42d9148ca80d4436a175e5b10" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } } peers { permanent_uuid: "d4f2c6961f124baebbeb52958b96c48f" member_type: NON_VOTER last_known_addr { host: "127.0.205.131" port: 41141 } attrs { promote: true } } }
I20260925 18:12:30.595459 5247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:30.600751 5098 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:30.606405 5173 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20260925 18:12:30.616954 5321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260925 18:12:30.793823 5588 ts_tablet_manager.cc:927] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: Initiating tablet copy from peer dada11d42d9148ca80d4436a175e5b10 (127.0.205.130:35655)
I20260925 18:12:30.795637 5588 tablet_copy_client.cc:316] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.205.130:35655
I20260925 18:12:30.807091 5193 tablet_copy_service.cc:140] P dada11d42d9148ca80d4436a175e5b10: Received BeginTabletCopySession request for tablet ba3f57e7cb32450db190b7448c5175ab from peer d4f2c6961f124baebbeb52958b96c48f ({username='slave'} at 127.0.0.1:41978)
I20260925 18:12:30.807642 5193 tablet_copy_service.cc:161] P dada11d42d9148ca80d4436a175e5b10: Beginning new tablet copy session on tablet ba3f57e7cb32450db190b7448c5175ab from peer d4f2c6961f124baebbeb52958b96c48f at {username='slave'} at 127.0.0.1:41978: session id = d4f2c6961f124baebbeb52958b96c48f-ba3f57e7cb32450db190b7448c5175ab
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
b22ef9812bc047579902b4223356005f | 127.0.205.190:34289 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
I20260925 18:12:30.814653 5193 tablet_copy_source_session.cc:205] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10: Tablet Copy: opened 0 blocks and 1 log segments
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0542a42032584d74ae62283be4cae3c5 | 127.0.205.129:41355 | HEALTHY | <none> | 0 | 0
d4f2c6961f124baebbeb52958b96c48f | 127.0.205.131:41141 | HEALTHY | <none> | 0 | 0
dada11d42d9148ca80d4436a175e5b10 | 127.0.205.130:35655 | HEALTHY | <none> | 1 | 0
f9dc8f9a62b742c0b03e47db27765ab8 | 127.0.205.132:40471 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
I20260925 18:12:30.817550 5588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3f57e7cb32450db190b7448c5175ab. 1 dirs total, 0 dirs full, 0 dirs failed
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | f9dc8f9a62b742c0b03e47db27765ab8 | 127.0.205.132:40471 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260925 18:12:30.824043 822 log_verifier.cc:126] Checking tablet ba3f57e7cb32450db190b7448c5175ab
I20260925 18:12:30.831955 5588 tablet_copy_client.cc:791] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: tablet copy: Starting download of 0 data blocks...
I20260925 18:12:30.832576 5588 tablet_copy_client.cc:655] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: tablet copy: Starting download of 1 WAL segments...
I20260925 18:12:30.836755 5588 tablet_copy_client.cc:531] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 18:12:30.843396 5588 tablet_bootstrap.cc:492] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: Bootstrap starting.
I20260925 18:12:30.863646 5588 tablet_bootstrap.cc:492] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 18:12:30.864302 5588 tablet_bootstrap.cc:492] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: Bootstrap complete.
I20260925 18:12:30.864804 5588 ts_tablet_manager.cc:1397] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.000s
I20260925 18:12:30.866846 5588 raft_consensus.cc:348] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } } peers { permanent_uuid: "d4f2c6961f124baebbeb52958b96c48f" member_type: NON_VOTER last_known_addr { host: "127.0.205.131" port: 41141 } attrs { promote: true } }
I20260925 18:12:30.867365 5588 raft_consensus.cc:729] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d4f2c6961f124baebbeb52958b96c48f, State: Initialized, Role: LEARNER
I20260925 18:12:30.867857 5588 consensus_queue.cc:260] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dada11d42d9148ca80d4436a175e5b10" member_type: VOTER last_known_addr { host: "127.0.205.130" port: 35655 } } peers { permanent_uuid: "0542a42032584d74ae62283be4cae3c5" member_type: VOTER last_known_addr { host: "127.0.205.129" port: 41355 } } peers { permanent_uuid: "d4f2c6961f124baebbeb52958b96c48f" member_type: NON_VOTER last_known_addr { host: "127.0.205.131" port: 41141 } attrs { promote: true } }
I20260925 18:12:30.870049 5588 ts_tablet_manager.cc:1428] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 18:12:30.871546 5193 tablet_copy_service.cc:342] P dada11d42d9148ca80d4436a175e5b10: Request end of tablet copy session d4f2c6961f124baebbeb52958b96c48f-ba3f57e7cb32450db190b7448c5175ab received from {username='slave'} at 127.0.0.1:41978
I20260925 18:12:30.872036 5193 tablet_copy_service.cc:433] P dada11d42d9148ca80d4436a175e5b10: ending tablet copy session d4f2c6961f124baebbeb52958b96c48f-ba3f57e7cb32450db190b7448c5175ab on tablet ba3f57e7cb32450db190b7448c5175ab with peer d4f2c6961f124baebbeb52958b96c48f
I20260925 18:12:30.893958 822 log_verifier.cc:177] Verified matching terms for 4 ops in tablet ba3f57e7cb32450db190b7448c5175ab
I20260925 18:12:30.896311 822 tablet_server.cc:178] TabletServer@127.0.205.129:0 shutting down...
I20260925 18:12:30.913692 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:30.914249 822 tablet_replica.cc:330] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5: stopping tablet replica
I20260925 18:12:30.914721 822 raft_consensus.cc:2229] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260925 18:12:30.915225 822 raft_consensus.cc:2258] T ba3f57e7cb32450db190b7448c5175ab P 0542a42032584d74ae62283be4cae3c5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:30.934922 822 tablet_server.cc:194] TabletServer@127.0.205.129:0 shutdown complete.
I20260925 18:12:30.943408 822 tablet_server.cc:178] TabletServer@127.0.205.130:0 shutting down...
I20260925 18:12:30.960480 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:30.961103 822 tablet_replica.cc:330] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10: stopping tablet replica
I20260925 18:12:30.961630 822 raft_consensus.cc:2229] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 2 LEADER]: Raft consensus shutting down.
I20260925 18:12:30.962412 822 raft_consensus.cc:2258] T ba3f57e7cb32450db190b7448c5175ab P dada11d42d9148ca80d4436a175e5b10 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:30.981665 822 tablet_server.cc:194] TabletServer@127.0.205.130:0 shutdown complete.
I20260925 18:12:30.989892 822 tablet_server.cc:178] TabletServer@127.0.205.131:0 shutting down...
I20260925 18:12:31.006039 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:31.006773 822 tablet_replica.cc:330] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f: stopping tablet replica
I20260925 18:12:31.007292 822 raft_consensus.cc:2229] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f [term 2 LEARNER]: Raft consensus shutting down.
I20260925 18:12:31.007663 822 raft_consensus.cc:2258] T ba3f57e7cb32450db190b7448c5175ab P d4f2c6961f124baebbeb52958b96c48f [term 2 LEARNER]: Raft consensus is shut down!
I20260925 18:12:31.026252 822 tablet_server.cc:194] TabletServer@127.0.205.131:0 shutdown complete.
I20260925 18:12:31.034735 822 tablet_server.cc:178] TabletServer@127.0.205.132:0 shutting down...
I20260925 18:12:31.050598 822 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260925 18:12:31.066987 822 tablet_server.cc:194] TabletServer@127.0.205.132:0 shutdown complete.
I20260925 18:12:31.075448 822 master.cc:537] Master@127.0.205.190:34289 shutting down...
I20260925 18:12:31.089248 822 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 1 LEADER]: Raft consensus shutting down.
I20260925 18:12:31.089742 822 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 18:12:31.090014 822 tablet_replica.cc:330] T 00000000000000000000000000000000 P b22ef9812bc047579902b4223356005f: stopping tablet replica
I20260925 18:12:31.107213 822 master.cc:552] Master@127.0.205.190:34289 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (4793 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (8430 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (69829 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20260925 18:12:31.140110 822 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~9 seconds ago
I20260925 18:12:31.140399 822 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 8 messages since previous log ~13 seconds ago
I20260925 18:12:31.140559 822 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tserver/heartbeater.cc:644: suppressed but not reported on 158 messages since previous log ~61 seconds ago
I20260925 18:12:31.140862 822 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tablet/tablet.cc:2369: suppressed but not reported on 8 messages since previous log ~12 seconds ago