[==========] Running 13 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from LeadershipChangeReportingTest
[ RUN ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20250903 02:09:37.866915 27791 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.150s user 0.441s sys 0.706s
W20250903 02:09:37.867640 27791 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.151s user 0.441s sys 0.706s
I20250903 02:09:37.870954 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:36475
I20250903 02:09:37.873543 27791 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 02:09:37.875553 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:37.892174 27802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:37.894181 27803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:38.048331 27805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:38.048697 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:38.049865 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:38.050143 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:38.050330 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865378050302 us; error 0 us; skew 500 ppm
I20250903 02:09:38.051071 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:38.055852 27791 webserver.cc:480] Webserver started at http://127.27.35.254:39871/ using document root <none> and password file <none>
I20250903 02:09:38.056703 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:38.056929 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:38.057355 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:38.062928 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "5b7897179bd24947a9003c00b8eb3462"
format_stamp: "Formatted at 2025-09-03 02:09:38 on dist-test-slave-h78g"
I20250903 02:09:38.070904 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250903 02:09:38.076417 27810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:38.077663 27791 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250903 02:09:38.078047 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "5b7897179bd24947a9003c00b8eb3462"
format_stamp: "Formatted at 2025-09-03 02:09:38 on dist-test-slave-h78g"
I20250903 02:09:38.078415 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:38.101362 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:38.102885 27791 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 02:09:38.103371 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:38.176018 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:36475
I20250903 02:09:38.176098 27861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:36475 every 8 connection(s)
I20250903 02:09:38.182344 27862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:38.205107 27862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462: Bootstrap starting.
I20250903 02:09:38.211413 27862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:38.213429 27862 log.cc:826] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462: Log is configured to *not* fsync() on all Append() calls
I20250903 02:09:38.218194 27862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462: No bootstrap required, opened a new log
I20250903 02:09:38.238663 27862 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7897179bd24947a9003c00b8eb3462" member_type: VOTER }
I20250903 02:09:38.239255 27862 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:38.239463 27862 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7897179bd24947a9003c00b8eb3462, State: Initialized, Role: FOLLOWER
I20250903 02:09:38.240311 27862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7897179bd24947a9003c00b8eb3462" member_type: VOTER }
I20250903 02:09:38.240808 27862 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:09:38.241079 27862 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:09:38.241364 27862 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:38.245508 27862 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7897179bd24947a9003c00b8eb3462" member_type: VOTER }
I20250903 02:09:38.246201 27862 leader_election.cc:304] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b7897179bd24947a9003c00b8eb3462; no voters:
I20250903 02:09:38.247946 27862 leader_election.cc:290] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:09:38.248361 27865 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:09:38.250980 27865 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 1 LEADER]: Becoming Leader. State: Replica: 5b7897179bd24947a9003c00b8eb3462, State: Running, Role: LEADER
I20250903 02:09:38.251930 27865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7897179bd24947a9003c00b8eb3462" member_type: VOTER }
I20250903 02:09:38.253026 27862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:09:38.259961 27866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b7897179bd24947a9003c00b8eb3462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7897179bd24947a9003c00b8eb3462" member_type: VOTER } }
I20250903 02:09:38.260154 27867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b7897179bd24947a9003c00b8eb3462. Latest consensus state: current_term: 1 leader_uuid: "5b7897179bd24947a9003c00b8eb3462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7897179bd24947a9003c00b8eb3462" member_type: VOTER } }
I20250903 02:09:38.260691 27866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:38.260843 27867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:38.264443 27873 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:09:38.275689 27873 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:09:38.281461 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:09:38.294394 27873 catalog_manager.cc:1349] Generated new cluster ID: 94b36aa9ebe745ec9d31e804ceee836a
I20250903 02:09:38.294719 27873 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:09:38.318478 27873 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:09:38.320081 27873 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:09:38.334620 27873 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462: Generated new TSK 0
I20250903 02:09:38.335610 27873 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:09:38.351414 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:38.360968 27884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:38.361961 27885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:38.364961 27791 server_base.cc:1047] running on GCE node
W20250903 02:09:38.364979 27887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:38.366062 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:38.366267 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:38.366431 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865378366408 us; error 0 us; skew 500 ppm
I20250903 02:09:38.366973 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:38.371423 27791 webserver.cc:480] Webserver started at http://127.27.35.193:37859/ using document root <none> and password file <none>
I20250903 02:09:38.371979 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:38.372190 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:38.372464 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:38.373687 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "83711a7203874bc79161fbca07f4dab9"
format_stamp: "Formatted at 2025-09-03 02:09:38 on dist-test-slave-h78g"
I20250903 02:09:38.379422 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250903 02:09:38.383905 27892 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:38.384706 27791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250903 02:09:38.384965 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "83711a7203874bc79161fbca07f4dab9"
format_stamp: "Formatted at 2025-09-03 02:09:38 on dist-test-slave-h78g"
I20250903 02:09:38.385242 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:38.414487 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:38.415817 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:38.418555 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:38.423209 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:38.423434 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:38.423727 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:38.423931 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250903 02:09:38.527935 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:40879
I20250903 02:09:38.528025 27954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:40879 every 8 connection(s)
I20250903 02:09:38.534909 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:38.545014 27959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:38.546231 27960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:38.550360 27791 server_base.cc:1047] running on GCE node
W20250903 02:09:38.551283 27963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:38.552482 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:38.552739 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:38.552915 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865378552899 us; error 0 us; skew 500 ppm
I20250903 02:09:38.553409 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:38.556824 27791 webserver.cc:480] Webserver started at http://127.27.35.194:44219/ using document root <none> and password file <none>
I20250903 02:09:38.557296 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:38.557466 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:38.557709 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:38.558804 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/ts-1-root/instance:
uuid: "c76e98e1eb944f94a4ee7b92c52acc87"
format_stamp: "Formatted at 2025-09-03 02:09:38 on dist-test-slave-h78g"
I20250903 02:09:38.564469 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20250903 02:09:38.568706 27967 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:38.569572 27791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250903 02:09:38.569893 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/ts-1-root
uuid: "c76e98e1eb944f94a4ee7b92c52acc87"
format_stamp: "Formatted at 2025-09-03 02:09:38 on dist-test-slave-h78g"
I20250903 02:09:38.570236 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestConcurrentGetTableLocations.1756865376507568-27791-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:38.587810 27955 heartbeater.cc:344] Connected to a master server at 127.27.35.254:36475
I20250903 02:09:38.588292 27955 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:38.589186 27955 heartbeater.cc:507] Master 127.27.35.254:36475 requested a full tablet report, sending...
I20250903 02:09:38.591921 27827 ts_manager.cc:194] Registered new tserver with Master: 83711a7203874bc79161fbca07f4dab9 (127.27.35.193:40879)
I20250903 02:09:38.594077 27827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38920
I20250903 02:09:38.613327 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:38.615046 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:38.617549 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:38.621141 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:38.621403 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:38.621688 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:38.621894 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:38.697228 27955 heartbeater.cc:499] Master 127.27.35.254:36475 was elected leader, sending a full tablet report...
I20250903 02:09:38.711409 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.194:36771
I20250903 02:09:38.711603 28029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.194:36771 every 8 connection(s)
I20250903 02:09:38.728148 28030 heartbeater.cc:344] Connected to a master server at 127.27.35.254:36475
I20250903 02:09:38.728492 28030 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:38.729214 28030 heartbeater.cc:507] Master 127.27.35.254:36475 requested a full tablet report, sending...
I20250903 02:09:38.730995 27827 ts_manager.cc:194] Registered new tserver with Master: c76e98e1eb944f94a4ee7b92c52acc87 (127.27.35.194:36771)
I20250903 02:09:38.731640 27791 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.014539037s
I20250903 02:09:38.732631 27827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38928
I20250903 02:09:38.762007 27827 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
range_schema {
columns {
name: "key"
}
}
}
W20250903 02:09:38.781054 27827 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20250903 02:09:38.829864 27919 tablet_service.cc:1468] Processing CreateTablet for tablet cb16509f2b084640936c3a030e2e9177 (DEFAULT_TABLE table=test-table [id=cd2f983526cc435693c1d8c69dcd0897]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:38.831568 27919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb16509f2b084640936c3a030e2e9177. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:38.829864 27920 tablet_service.cc:1468] Processing CreateTablet for tablet c3ebc6d8acb94915816077b5a59c329b (DEFAULT_TABLE table=test-table [id=cd2f983526cc435693c1d8c69dcd0897]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:38.834347 27920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3ebc6d8acb94915816077b5a59c329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:38.835680 28030 heartbeater.cc:499] Master 127.27.35.254:36475 was elected leader, sending a full tablet report...
I20250903 02:09:38.840441 27995 tablet_service.cc:1468] Processing CreateTablet for tablet c3ebc6d8acb94915816077b5a59c329b (DEFAULT_TABLE table=test-table [id=cd2f983526cc435693c1d8c69dcd0897]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:38.840819 27994 tablet_service.cc:1468] Processing CreateTablet for tablet cb16509f2b084640936c3a030e2e9177 (DEFAULT_TABLE table=test-table [id=cd2f983526cc435693c1d8c69dcd0897]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:38.841670 27995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3ebc6d8acb94915816077b5a59c329b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:38.842339 27994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb16509f2b084640936c3a030e2e9177. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:38.862527 28043 tablet_bootstrap.cc:492] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9: Bootstrap starting.
I20250903 02:09:38.862535 28044 tablet_bootstrap.cc:492] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87: Bootstrap starting.
I20250903 02:09:38.871893 28043 tablet_bootstrap.cc:654] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:38.871897 28044 tablet_bootstrap.cc:654] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:38.879689 28043 tablet_bootstrap.cc:492] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9: No bootstrap required, opened a new log
I20250903 02:09:38.879792 28044 tablet_bootstrap.cc:492] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87: No bootstrap required, opened a new log
I20250903 02:09:38.880283 28043 ts_tablet_manager.cc:1397] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20250903 02:09:38.880306 28044 ts_tablet_manager.cc:1397] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20250903 02:09:38.883080 28044 raft_consensus.cc:357] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:38.883606 28044 raft_consensus.cc:738] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76e98e1eb944f94a4ee7b92c52acc87, State: Initialized, Role: FOLLOWER
I20250903 02:09:38.883365 28043 raft_consensus.cc:357] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:38.884064 28043 raft_consensus.cc:738] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83711a7203874bc79161fbca07f4dab9, State: Initialized, Role: FOLLOWER
I20250903 02:09:38.884189 28044 consensus_queue.cc:260] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:38.884675 28043 consensus_queue.cc:260] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:38.887023 28044 ts_tablet_manager.cc:1428] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250903 02:09:38.888104 28044 tablet_bootstrap.cc:492] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87: Bootstrap starting.
I20250903 02:09:38.894665 28044 tablet_bootstrap.cc:654] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:38.897766 28043 ts_tablet_manager.cc:1428] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
I20250903 02:09:38.901872 28044 tablet_bootstrap.cc:492] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87: No bootstrap required, opened a new log
I20250903 02:09:38.902367 28044 ts_tablet_manager.cc:1397] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20250903 02:09:38.904956 28044 raft_consensus.cc:357] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:38.905684 28044 raft_consensus.cc:738] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76e98e1eb944f94a4ee7b92c52acc87, State: Initialized, Role: FOLLOWER
I20250903 02:09:38.906541 28043 tablet_bootstrap.cc:492] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9: Bootstrap starting.
I20250903 02:09:38.906272 28044 consensus_queue.cc:260] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:38.908952 28044 ts_tablet_manager.cc:1428] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250903 02:09:38.913087 28043 tablet_bootstrap.cc:654] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:38.919500 28043 tablet_bootstrap.cc:492] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9: No bootstrap required, opened a new log
I20250903 02:09:38.920002 28043 ts_tablet_manager.cc:1397] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20250903 02:09:38.922616 28043 raft_consensus.cc:357] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:38.923286 28043 raft_consensus.cc:738] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83711a7203874bc79161fbca07f4dab9, State: Initialized, Role: FOLLOWER
I20250903 02:09:38.923885 28043 consensus_queue.cc:260] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:38.926203 28043 ts_tablet_manager.cc:1428] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250903 02:09:38.958220 27791 ts_tablet_manager-itest.cc:368] Electing peer 2...
I20250903 02:09:38.958568 27791 raft_consensus.cc:421] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 0 FOLLOWER]: Emulating election...
I20250903 02:09:38.958818 27791 raft_consensus.cc:3058] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:38.962808 27791 raft_consensus.cc:695] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 1 LEADER]: Becoming Leader. State: Replica: c76e98e1eb944f94a4ee7b92c52acc87, State: Running, Role: LEADER
I20250903 02:09:38.963524 27791 consensus_queue.cc:237] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:38.967516 27791 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
W20250903 02:09:38.968760 28031 tablet.cc:2378] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 02:09:38.975227 27826 catalog_manager.cc:5582] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 reported cstate change: term changed from 0 to 1, leader changed from <none> to c76e98e1eb944f94a4ee7b92c52acc87 (127.27.35.194). New cstate: current_term: 1 leader_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } health_report { overall_health: HEALTHY } } }
I20250903 02:09:38.995445 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250903 02:09:39.099475 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250903 02:09:39.303617 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250903 02:09:39.359709 27930 raft_consensus.cc:3058] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:39.366670 28047 consensus_queue.cc:1035] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250903 02:09:39.607790 27791 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20250903 02:09:39.608058 27791 raft_consensus.cc:421] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 0 FOLLOWER]: Emulating election...
I20250903 02:09:39.608273 27791 raft_consensus.cc:3058] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:39.612020 27791 raft_consensus.cc:695] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 1 LEADER]: Becoming Leader. State: Replica: c76e98e1eb944f94a4ee7b92c52acc87, State: Running, Role: LEADER
I20250903 02:09:39.612653 27791 consensus_queue.cc:237] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:39.617192 27791 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250903 02:09:39.619511 27826 catalog_manager.cc:5582] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 reported cstate change: term changed from 0 to 1, leader changed from <none> to c76e98e1eb944f94a4ee7b92c52acc87 (127.27.35.194). New cstate: current_term: 1 leader_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } health_report { overall_health: HEALTHY } } }
I20250903 02:09:39.620913 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250903 02:09:39.724758 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250903 02:09:39.928464 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 0.0 1.1
I20250903 02:09:39.994482 27930 raft_consensus.cc:3058] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:40.000936 28047 consensus_queue.cc:1035] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 02:09:40.432266 27791 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20250903 02:09:40.432579 27791 raft_consensus.cc:421] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 1 LEADER]: Emulating election...
I20250903 02:09:40.432849 27791 raft_consensus.cc:3053] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 02:09:40.433118 27791 raft_consensus.cc:738] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c76e98e1eb944f94a4ee7b92c52acc87, State: Running, Role: LEADER
I20250903 02:09:40.433683 27791 consensus_queue.cc:260] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:40.434593 27791 raft_consensus.cc:3058] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:40.439389 27791 raft_consensus.cc:695] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 2 LEADER]: Becoming Leader. State: Replica: c76e98e1eb944f94a4ee7b92c52acc87, State: Running, Role: LEADER
I20250903 02:09:40.440028 27791 consensus_queue.cc:237] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:40.441375 27791 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250903 02:09:40.445083 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 1.1 2.2
I20250903 02:09:40.444972 27825 catalog_manager.cc:5582] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } health_report { overall_health: HEALTHY } } }
I20250903 02:09:40.549350 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 1.1 2.2
I20250903 02:09:40.753558 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 1.1 2.2
I20250903 02:09:40.934650 27930 raft_consensus.cc:3058] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:40.939610 27930 raft_consensus.cc:1273] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 2 FOLLOWER]: Refusing update from remote peer c76e98e1eb944f94a4ee7b92c52acc87: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250903 02:09:40.940866 28065 consensus_queue.cc:1035] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250903 02:09:41.058497 27791 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20250903 02:09:41.058804 27791 raft_consensus.cc:421] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 2 FOLLOWER]: Emulating election...
I20250903 02:09:41.059082 27791 raft_consensus.cc:3058] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:41.065460 27791 raft_consensus.cc:695] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 3 LEADER]: Becoming Leader. State: Replica: 83711a7203874bc79161fbca07f4dab9, State: Running, Role: LEADER
I20250903 02:09:41.066067 27791 consensus_queue.cc:237] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:41.067700 27791 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250903 02:09:41.071843 27825 catalog_manager.cc:5582] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 reported cstate change: term changed from 2 to 3, leader changed from c76e98e1eb944f94a4ee7b92c52acc87 (127.27.35.194) to 83711a7203874bc79161fbca07f4dab9 (127.27.35.193). New cstate: current_term: 3 leader_uuid: "83711a7203874bc79161fbca07f4dab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } health_report { overall_health: UNKNOWN } } }
I20250903 02:09:41.073206 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20250903 02:09:41.176726 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20250903 02:09:41.380843 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 2.2
I20250903 02:09:41.424055 27930 raft_consensus.cc:1238] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 3 LEADER]: Rejecting Update request from peer c76e98e1eb944f94a4ee7b92c52acc87 for earlier term 2. Current term is 3. Ops: []
I20250903 02:09:41.425268 28072 consensus_queue.cc:1046] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250903 02:09:41.425894 28072 consensus_queue.cc:1230] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Peer 83711a7203874bc79161fbca07f4dab9 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250903 02:09:41.426367 28065 raft_consensus.cc:3053] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 2 LEADER]: Stepping down as leader of term 2
I20250903 02:09:41.426627 28065 raft_consensus.cc:738] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c76e98e1eb944f94a4ee7b92c52acc87, State: Running, Role: LEADER
I20250903 02:09:41.427165 28065 consensus_queue.cc:260] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:41.427893 28065 raft_consensus.cc:3058] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:41.663448 28005 raft_consensus.cc:1273] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 3 FOLLOWER]: Refusing update from remote peer 83711a7203874bc79161fbca07f4dab9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250903 02:09:41.664875 28078 consensus_queue.cc:1035] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250903 02:09:41.884855 27791 ts_tablet_manager-itest.cc:368] Electing peer 1...
I20250903 02:09:41.885193 27791 raft_consensus.cc:421] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [term 1 FOLLOWER]: Emulating election...
I20250903 02:09:41.885519 27791 raft_consensus.cc:3058] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:41.890038 27791 raft_consensus.cc:695] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [term 2 LEADER]: Becoming Leader. State: Replica: 83711a7203874bc79161fbca07f4dab9, State: Running, Role: LEADER
I20250903 02:09:41.890589 27791 consensus_queue.cc:237] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:41.892613 27791 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250903 02:09:41.896162 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20250903 02:09:41.896258 27825 catalog_manager.cc:5582] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 reported cstate change: term changed from 1 to 2, leader changed from c76e98e1eb944f94a4ee7b92c52acc87 (127.27.35.194) to 83711a7203874bc79161fbca07f4dab9 (127.27.35.193). New cstate: current_term: 2 leader_uuid: "83711a7203874bc79161fbca07f4dab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } health_report { overall_health: UNKNOWN } } }
I20250903 02:09:42.000371 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20250903 02:09:42.047731 27930 raft_consensus.cc:1238] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [term 2 LEADER]: Rejecting Update request from peer c76e98e1eb944f94a4ee7b92c52acc87 for earlier term 1. Current term is 2. Ops: []
I20250903 02:09:42.048992 28065 consensus_queue.cc:1046] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250903 02:09:42.050458 28065 consensus_queue.cc:1230] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Peer 83711a7203874bc79161fbca07f4dab9 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250903 02:09:42.050899 28088 raft_consensus.cc:3053] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 02:09:42.051175 28088 raft_consensus.cc:738] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c76e98e1eb944f94a4ee7b92c52acc87, State: Running, Role: LEADER
I20250903 02:09:42.051786 28088 consensus_queue.cc:260] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:42.052677 28088 raft_consensus.cc:3058] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:42.204262 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 2.2 1.1
I20250903 02:09:42.330622 28005 raft_consensus.cc:1273] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 2 FOLLOWER]: Refusing update from remote peer 83711a7203874bc79161fbca07f4dab9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250903 02:09:42.331811 28081 consensus_queue.cc:1035] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250903 02:09:42.508752 27791 ts_tablet_manager-itest.cc:368] Electing peer 3...
I20250903 02:09:42.509008 27791 raft_consensus.cc:421] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 3 FOLLOWER]: Emulating election...
I20250903 02:09:42.509243 27791 raft_consensus.cc:3058] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:09:42.513537 27791 raft_consensus.cc:695] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 4 LEADER]: Becoming Leader. State: Replica: c76e98e1eb944f94a4ee7b92c52acc87, State: Running, Role: LEADER
I20250903 02:09:42.514268 27791 consensus_queue.cc:237] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:42.515669 27791 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250903 02:09:42.519413 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20250903 02:09:42.519169 27825 catalog_manager.cc:5582] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 reported cstate change: term changed from 3 to 4, leader changed from 83711a7203874bc79161fbca07f4dab9 (127.27.35.193) to c76e98e1eb944f94a4ee7b92c52acc87 (127.27.35.194). New cstate: current_term: 4 leader_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } health_report { overall_health: HEALTHY } } }
I20250903 02:09:42.594703 28005 raft_consensus.cc:1238] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 4 LEADER]: Rejecting Update request from peer 83711a7203874bc79161fbca07f4dab9 for earlier term 3. Current term is 4. Ops: []
I20250903 02:09:42.596119 28090 consensus_queue.cc:1046] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250903 02:09:42.596740 28090 consensus_queue.cc:1230] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [LEADER]: Peer c76e98e1eb944f94a4ee7b92c52acc87 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250903 02:09:42.597072 28081 raft_consensus.cc:3053] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 3 LEADER]: Stepping down as leader of term 3
I20250903 02:09:42.597461 28081 raft_consensus.cc:738] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83711a7203874bc79161fbca07f4dab9, State: Running, Role: LEADER
I20250903 02:09:42.597898 28081 consensus_queue.cc:260] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 } } peers { permanent_uuid: "c76e98e1eb944f94a4ee7b92c52acc87" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 36771 } }
I20250903 02:09:42.598557 28081 raft_consensus.cc:3058] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:09:42.623149 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20250903 02:09:42.826905 27791 cluster_itest_util.cc:257] Not converged past 0 yet: 3.3 4.4
I20250903 02:09:42.977535 27930 raft_consensus.cc:1273] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 4 FOLLOWER]: Refusing update from remote peer c76e98e1eb944f94a4ee7b92c52acc87: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250903 02:09:42.978674 28088 consensus_queue.cc:1035] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83711a7203874bc79161fbca07f4dab9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250903 02:09:43.332975 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:09:43.353346 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:43.353950 27791 tablet_replica.cc:331] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9: stopping tablet replica
I20250903 02:09:43.354534 27791 raft_consensus.cc:2241] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250903 02:09:43.354960 27791 raft_consensus.cc:2270] T c3ebc6d8acb94915816077b5a59c329b P 83711a7203874bc79161fbca07f4dab9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:43.357339 27791 tablet_replica.cc:331] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9: stopping tablet replica
I20250903 02:09:43.357858 27791 raft_consensus.cc:2241] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [term 2 LEADER]: Raft consensus shutting down.
I20250903 02:09:43.358309 27791 raft_consensus.cc:2270] T cb16509f2b084640936c3a030e2e9177 P 83711a7203874bc79161fbca07f4dab9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:43.442211 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:09:43.451076 27791 tablet_server.cc:178] TabletServer@127.27.35.194:0 shutting down...
W20250903 02:09:43.465059 27969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.35.193:40879: connect: Connection refused (error 111)
I20250903 02:09:43.468688 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:43.469270 27791 tablet_replica.cc:331] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87: stopping tablet replica
W20250903 02:09:43.469642 27969 consensus_peers.cc:489] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 -> Peer 83711a7203874bc79161fbca07f4dab9 (127.27.35.193:40879): Couldn't send request to peer 83711a7203874bc79161fbca07f4dab9. Status: Network error: Client connection negotiation failed: client connection to 127.27.35.193:40879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 02:09:43.469998 27791 raft_consensus.cc:2241] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250903 02:09:43.470432 27791 raft_consensus.cc:2270] T cb16509f2b084640936c3a030e2e9177 P c76e98e1eb944f94a4ee7b92c52acc87 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:43.471858 27791 tablet_replica.cc:331] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87: stopping tablet replica
I20250903 02:09:43.472329 27791 raft_consensus.cc:2241] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 4 LEADER]: Raft consensus shutting down.
I20250903 02:09:43.472874 27791 raft_consensus.cc:2270] T c3ebc6d8acb94915816077b5a59c329b P c76e98e1eb944f94a4ee7b92c52acc87 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:43.491696 27791 tablet_server.cc:195] TabletServer@127.27.35.194:0 shutdown complete.
I20250903 02:09:43.500115 27791 master.cc:561] Master@127.27.35.254:36475 shutting down...
I20250903 02:09:43.692873 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:09:43.693359 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:43.693617 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5b7897179bd24947a9003c00b8eb3462: stopping tablet replica
I20250903 02:09:43.712302 27791 master.cc:583] Master@127.27.35.254:36475 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestConcurrentGetTableLocations (7063 ms)
[ RUN ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
I20250903 02:09:43.737627 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:35329
I20250903 02:09:43.738605 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:43.743398 28106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:43.744459 28107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:43.747005 27791 server_base.cc:1047] running on GCE node
W20250903 02:09:43.746914 28109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:43.749266 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:43.749517 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:43.749688 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865383749670 us; error 0 us; skew 500 ppm
I20250903 02:09:43.750180 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:43.752502 27791 webserver.cc:480] Webserver started at http://127.27.35.254:38087/ using document root <none> and password file <none>
I20250903 02:09:43.752948 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:43.753141 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:43.753389 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:43.754436 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "6214027b41814a64997a80ccaacb1c8d"
format_stamp: "Formatted at 2025-09-03 02:09:43 on dist-test-slave-h78g"
I20250903 02:09:43.758620 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:09:43.761839 28114 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:43.762647 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 02:09:43.762888 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "6214027b41814a64997a80ccaacb1c8d"
format_stamp: "Formatted at 2025-09-03 02:09:43 on dist-test-slave-h78g"
I20250903 02:09:43.763139 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:43.795320 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:43.796283 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:43.830902 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:35329
I20250903 02:09:43.831027 28165 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:35329 every 8 connection(s)
I20250903 02:09:43.835003 28166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:43.844659 28166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d: Bootstrap starting.
I20250903 02:09:43.848942 28166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:43.853034 28166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d: No bootstrap required, opened a new log
I20250903 02:09:43.854903 28166 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6214027b41814a64997a80ccaacb1c8d" member_type: VOTER }
I20250903 02:09:43.855301 28166 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:43.855556 28166 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6214027b41814a64997a80ccaacb1c8d, State: Initialized, Role: FOLLOWER
I20250903 02:09:43.856187 28166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6214027b41814a64997a80ccaacb1c8d" member_type: VOTER }
I20250903 02:09:43.856609 28166 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:09:43.856846 28166 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:09:43.857113 28166 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:43.861183 28166 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6214027b41814a64997a80ccaacb1c8d" member_type: VOTER }
I20250903 02:09:43.861743 28166 leader_election.cc:304] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6214027b41814a64997a80ccaacb1c8d; no voters:
I20250903 02:09:43.862841 28166 leader_election.cc:290] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:09:43.863153 28169 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:09:43.864552 28169 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 1 LEADER]: Becoming Leader. State: Replica: 6214027b41814a64997a80ccaacb1c8d, State: Running, Role: LEADER
I20250903 02:09:43.865278 28169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6214027b41814a64997a80ccaacb1c8d" member_type: VOTER }
I20250903 02:09:43.865880 28166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:09:43.868152 28171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6214027b41814a64997a80ccaacb1c8d. Latest consensus state: current_term: 1 leader_uuid: "6214027b41814a64997a80ccaacb1c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6214027b41814a64997a80ccaacb1c8d" member_type: VOTER } }
I20250903 02:09:43.868046 28170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6214027b41814a64997a80ccaacb1c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6214027b41814a64997a80ccaacb1c8d" member_type: VOTER } }
I20250903 02:09:43.868664 28171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:43.868858 28170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:43.874064 28174 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:09:43.881778 28174 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:09:43.882771 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:09:43.890450 28174 catalog_manager.cc:1349] Generated new cluster ID: c3b3d164584b4f679337594787e66a71
I20250903 02:09:43.890756 28174 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:09:43.914148 28174 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:09:43.915319 28174 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:09:43.926692 28174 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d: Generated new TSK 0
I20250903 02:09:43.927286 28174 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:09:43.949151 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:43.955240 28188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:43.955459 28187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:43.957111 28190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:43.957769 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:43.958505 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:43.958678 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:43.958859 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865383958840 us; error 0 us; skew 500 ppm
I20250903 02:09:43.959363 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:43.961625 27791 webserver.cc:480] Webserver started at http://127.27.35.193:34987/ using document root <none> and password file <none>
I20250903 02:09:43.962064 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:43.962221 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:43.962467 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:43.963454 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "bb7dd437228e4ab7b83c06c1ba9ab03b"
format_stamp: "Formatted at 2025-09-03 02:09:43 on dist-test-slave-h78g"
I20250903 02:09:43.967898 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250903 02:09:43.970944 28195 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:43.971674 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:09:43.971997 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "bb7dd437228e4ab7b83c06c1ba9ab03b"
format_stamp: "Formatted at 2025-09-03 02:09:43 on dist-test-slave-h78g"
I20250903 02:09:43.972270 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:43.985592 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:43.986521 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:43.987856 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:43.989852 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:43.990033 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:43.990252 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:43.990406 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.026094 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:39017
I20250903 02:09:44.026190 28257 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:39017 every 8 connection(s)
I20250903 02:09:44.030113 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:44.036996 28262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:44.038664 28263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:44.040158 28265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:44.041020 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:44.041747 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:44.041944 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:44.042099 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865384042087 us; error 0 us; skew 500 ppm
I20250903 02:09:44.042527 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:44.045239 27791 webserver.cc:480] Webserver started at http://127.27.35.194:41557/ using document root <none> and password file <none>
I20250903 02:09:44.045897 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:44.046124 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:44.046036 28258 heartbeater.cc:344] Connected to a master server at 127.27.35.254:35329
I20250903 02:09:44.046489 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:44.046573 28258 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:44.047312 28258 heartbeater.cc:507] Master 127.27.35.254:35329 requested a full tablet report, sending...
I20250903 02:09:44.047523 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/ts-1-root/instance:
uuid: "f0daf377e86f4631b30cb737f238b7e4"
format_stamp: "Formatted at 2025-09-03 02:09:44 on dist-test-slave-h78g"
I20250903 02:09:44.049283 28131 ts_manager.cc:194] Registered new tserver with Master: bb7dd437228e4ab7b83c06c1ba9ab03b (127.27.35.193:39017)
I20250903 02:09:44.050868 28131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52982
I20250903 02:09:44.052412 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250903 02:09:44.055572 28270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.056324 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 02:09:44.056595 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/ts-1-root
uuid: "f0daf377e86f4631b30cb737f238b7e4"
format_stamp: "Formatted at 2025-09-03 02:09:44 on dist-test-slave-h78g"
I20250903 02:09:44.056865 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange.1756865376507568-27791-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:44.069837 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:44.070878 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:44.072281 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:44.074587 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:44.074764 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.074987 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:44.075129 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.123591 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.194:34523
I20250903 02:09:44.123673 28332 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.194:34523 every 8 connection(s)
I20250903 02:09:44.136479 28333 heartbeater.cc:344] Connected to a master server at 127.27.35.254:35329
I20250903 02:09:44.136828 28333 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:44.137518 28333 heartbeater.cc:507] Master 127.27.35.254:35329 requested a full tablet report, sending...
I20250903 02:09:44.139313 28131 ts_manager.cc:194] Registered new tserver with Master: f0daf377e86f4631b30cb737f238b7e4 (127.27.35.194:34523)
I20250903 02:09:44.139961 27791 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.013261596s
I20250903 02:09:44.140636 28131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52986
I20250903 02:09:44.153324 28258 heartbeater.cc:499] Master 127.27.35.254:35329 was elected leader, sending a full tablet report...
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:422: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
I20250903 02:09:44.161688 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:09:44.175860 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:44.190984 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:09:44.197625 27791 tablet_server.cc:178] TabletServer@127.27.35.194:0 shutting down...
I20250903 02:09:44.212306 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:44.228096 27791 tablet_server.cc:195] TabletServer@127.27.35.194:0 shutdown complete.
I20250903 02:09:44.234525 27791 master.cc:561] Master@127.27.35.254:35329 shutting down...
I20250903 02:09:44.247594 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:09:44.248180 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:44.248574 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6214027b41814a64997a80ccaacb1c8d: stopping tablet replica
I20250903 02:09:44.266832 27791 master.cc:583] Master@127.27.35.254:35329 shutdown complete.
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange (550 ms)
[ RUN ] LeadershipChangeReportingTest.TestUpdatedConsensusState
I20250903 02:09:44.287585 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:43613
I20250903 02:09:44.288597 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:44.293159 28346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:44.294986 28347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:44.295820 28349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:44.296185 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:44.296898 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:44.297070 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:44.297186 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865384297175 us; error 0 us; skew 500 ppm
I20250903 02:09:44.297624 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:44.299938 27791 webserver.cc:480] Webserver started at http://127.27.35.254:42957/ using document root <none> and password file <none>
I20250903 02:09:44.300380 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:44.300524 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:44.300731 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:44.301692 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "7884271c7d804b7cbbdd4e76792056be"
format_stamp: "Formatted at 2025-09-03 02:09:44 on dist-test-slave-h78g"
I20250903 02:09:44.305940 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:09:44.309127 28354 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.309928 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:09:44.310246 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "7884271c7d804b7cbbdd4e76792056be"
format_stamp: "Formatted at 2025-09-03 02:09:44 on dist-test-slave-h78g"
I20250903 02:09:44.310531 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:44.321509 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:44.322499 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:44.356235 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:43613
I20250903 02:09:44.356369 28405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:43613 every 8 connection(s)
I20250903 02:09:44.359898 28406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:44.369175 28406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be: Bootstrap starting.
I20250903 02:09:44.373154 28406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:44.376871 28406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be: No bootstrap required, opened a new log
I20250903 02:09:44.378643 28406 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7884271c7d804b7cbbdd4e76792056be" member_type: VOTER }
I20250903 02:09:44.379086 28406 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:44.379268 28406 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7884271c7d804b7cbbdd4e76792056be, State: Initialized, Role: FOLLOWER
I20250903 02:09:44.379789 28406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7884271c7d804b7cbbdd4e76792056be" member_type: VOTER }
I20250903 02:09:44.380188 28406 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:09:44.380378 28406 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:09:44.380584 28406 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:44.384524 28406 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7884271c7d804b7cbbdd4e76792056be" member_type: VOTER }
I20250903 02:09:44.384990 28406 leader_election.cc:304] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7884271c7d804b7cbbdd4e76792056be; no voters:
I20250903 02:09:44.386132 28406 leader_election.cc:290] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:09:44.386415 28409 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:09:44.387692 28409 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 1 LEADER]: Becoming Leader. State: Replica: 7884271c7d804b7cbbdd4e76792056be, State: Running, Role: LEADER
I20250903 02:09:44.388447 28409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7884271c7d804b7cbbdd4e76792056be" member_type: VOTER }
I20250903 02:09:44.389983 28406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:09:44.391301 28411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7884271c7d804b7cbbdd4e76792056be. Latest consensus state: current_term: 1 leader_uuid: "7884271c7d804b7cbbdd4e76792056be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7884271c7d804b7cbbdd4e76792056be" member_type: VOTER } }
I20250903 02:09:44.391381 28410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7884271c7d804b7cbbdd4e76792056be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7884271c7d804b7cbbdd4e76792056be" member_type: VOTER } }
I20250903 02:09:44.392045 28411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:44.392112 28410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:44.401006 28414 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:09:44.405701 28414 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:09:44.406610 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:09:44.413743 28414 catalog_manager.cc:1349] Generated new cluster ID: 220df33825fa41b681ff8b61cc998ea5
I20250903 02:09:44.414016 28414 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:09:44.434836 28414 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:09:44.436036 28414 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:09:44.457504 28414 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be: Generated new TSK 0
I20250903 02:09:44.458089 28414 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:09:44.472934 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:44.478360 28427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:44.479727 28428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:44.481307 27791 server_base.cc:1047] running on GCE node
W20250903 02:09:44.482045 28430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:44.482820 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:44.483006 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:44.483187 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865384483167 us; error 0 us; skew 500 ppm
I20250903 02:09:44.483685 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:44.485988 27791 webserver.cc:480] Webserver started at http://127.27.35.193:34247/ using document root <none> and password file <none>
I20250903 02:09:44.486428 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:44.486583 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:44.486835 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:44.487875 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "d9545a5d76fe4aaea54dc551a51e1967"
format_stamp: "Formatted at 2025-09-03 02:09:44 on dist-test-slave-h78g"
I20250903 02:09:44.492148 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250903 02:09:44.495273 28435 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.495975 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 02:09:44.496232 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "d9545a5d76fe4aaea54dc551a51e1967"
format_stamp: "Formatted at 2025-09-03 02:09:44 on dist-test-slave-h78g"
I20250903 02:09:44.496567 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:44.504760 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:44.505757 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:44.507074 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:44.509150 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:44.509343 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.509572 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:44.509719 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.545913 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:41651
I20250903 02:09:44.546008 28497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:41651 every 8 connection(s)
I20250903 02:09:44.550196 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:44.557457 28502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:44.557696 28503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:44.561721 28505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:44.561976 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:44.562644 28498 heartbeater.cc:344] Connected to a master server at 127.27.35.254:43613
I20250903 02:09:44.562839 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250903 02:09:44.562959 28498 heartbeater.cc:461] Registering TS with master...
W20250903 02:09:44.563071 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:44.563263 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865384563246 us; error 0 us; skew 500 ppm
I20250903 02:09:44.563717 28498 heartbeater.cc:507] Master 127.27.35.254:43613 requested a full tablet report, sending...
I20250903 02:09:44.563825 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:44.565873 28371 ts_manager.cc:194] Registered new tserver with Master: d9545a5d76fe4aaea54dc551a51e1967 (127.27.35.193:41651)
I20250903 02:09:44.566745 27791 webserver.cc:480] Webserver started at http://127.27.35.194:36393/ using document root <none> and password file <none>
I20250903 02:09:44.567356 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:44.567595 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:44.567926 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:44.568252 28371 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53508
I20250903 02:09:44.569319 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/ts-1-root/instance:
uuid: "cb60c8a8036c425a97155af7f345ea04"
format_stamp: "Formatted at 2025-09-03 02:09:44 on dist-test-slave-h78g"
I20250903 02:09:44.573609 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:09:44.576790 28510 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.577525 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:09:44.577783 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/ts-1-root
uuid: "cb60c8a8036c425a97155af7f345ea04"
format_stamp: "Formatted at 2025-09-03 02:09:44 on dist-test-slave-h78g"
I20250903 02:09:44.578027 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.LeadershipChangeReportingTest.TestUpdatedConsensusState.1756865376507568-27791-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:44.622853 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:44.623890 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:44.625268 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:44.627440 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:44.627635 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.627880 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:44.628031 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:44.664098 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.194:43753
I20250903 02:09:44.664175 28572 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.194:43753 every 8 connection(s)
I20250903 02:09:44.670730 28498 heartbeater.cc:499] Master 127.27.35.254:43613 was elected leader, sending a full tablet report...
I20250903 02:09:44.677371 28573 heartbeater.cc:344] Connected to a master server at 127.27.35.254:43613
I20250903 02:09:44.677685 28573 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:44.678308 28573 heartbeater.cc:507] Master 127.27.35.254:43613 requested a full tablet report, sending...
I20250903 02:09:44.680156 28371 ts_manager.cc:194] Registered new tserver with Master: cb60c8a8036c425a97155af7f345ea04 (127.27.35.194:43753)
I20250903 02:09:44.680348 27791 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.013208773s
I20250903 02:09:44.681433 28371 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53510
I20250903 02:09:44.701503 28371 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 02:09:44.702858 28371 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20250903 02:09:44.731611 28463 tablet_service.cc:1468] Processing CreateTablet for tablet d99684da8a334346838084271c83aa52 (DEFAULT_TABLE table=test-table [id=aee8f4c1f8d9474fa0ed80e16ce99304]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:44.732837 28463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d99684da8a334346838084271c83aa52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:44.733479 28538 tablet_service.cc:1468] Processing CreateTablet for tablet d99684da8a334346838084271c83aa52 (DEFAULT_TABLE table=test-table [id=aee8f4c1f8d9474fa0ed80e16ce99304]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:44.734556 28538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d99684da8a334346838084271c83aa52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:44.745250 28586 tablet_bootstrap.cc:492] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967: Bootstrap starting.
I20250903 02:09:44.746119 28587 tablet_bootstrap.cc:492] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04: Bootstrap starting.
I20250903 02:09:44.752067 28587 tablet_bootstrap.cc:654] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:44.752067 28586 tablet_bootstrap.cc:654] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:44.757477 28587 tablet_bootstrap.cc:492] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04: No bootstrap required, opened a new log
I20250903 02:09:44.757946 28587 ts_tablet_manager.cc:1397] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250903 02:09:44.758472 28586 tablet_bootstrap.cc:492] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967: No bootstrap required, opened a new log
I20250903 02:09:44.758816 28586 ts_tablet_manager.cc:1397] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20250903 02:09:44.759949 28587 raft_consensus.cc:357] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9545a5d76fe4aaea54dc551a51e1967" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41651 } } peers { permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 } }
I20250903 02:09:44.760417 28587 raft_consensus.cc:738] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb60c8a8036c425a97155af7f345ea04, State: Initialized, Role: FOLLOWER
I20250903 02:09:44.760689 28586 raft_consensus.cc:357] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9545a5d76fe4aaea54dc551a51e1967" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41651 } } peers { permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 } }
I20250903 02:09:44.760864 28587 consensus_queue.cc:260] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9545a5d76fe4aaea54dc551a51e1967" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41651 } } peers { permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 } }
I20250903 02:09:44.761152 28586 raft_consensus.cc:738] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9545a5d76fe4aaea54dc551a51e1967, State: Initialized, Role: FOLLOWER
I20250903 02:09:44.761583 28586 consensus_queue.cc:260] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9545a5d76fe4aaea54dc551a51e1967" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41651 } } peers { permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 } }
I20250903 02:09:44.763087 28573 heartbeater.cc:499] Master 127.27.35.254:43613 was elected leader, sending a full tablet report...
I20250903 02:09:44.763432 28587 ts_tablet_manager.cc:1428] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250903 02:09:44.765291 28586 ts_tablet_manager.cc:1428] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20250903 02:09:44.796013 27791 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20250903 02:09:44.796226 27791 raft_consensus.cc:421] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 0 FOLLOWER]: Emulating election...
I20250903 02:09:44.796463 27791 raft_consensus.cc:3058] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:44.800580 27791 raft_consensus.cc:695] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 1 LEADER]: Becoming Leader. State: Replica: d9545a5d76fe4aaea54dc551a51e1967, State: Running, Role: LEADER
W20250903 02:09:44.800928 28498 heartbeater.cc:646] Failed to heartbeat to 127.27.35.254:43613 (0 consecutive failures): IO error: failing all heartbeats for tests
I20250903 02:09:44.801237 27791 consensus_queue.cc:237] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9545a5d76fe4aaea54dc551a51e1967" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41651 } } peers { permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 } }
I20250903 02:09:44.803961 27791 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250903 02:09:44.821822 27791 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0
I20250903 02:09:44.925962 27791 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0
I20250903 02:09:45.130379 27791 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0
I20250903 02:09:45.249764 28548 raft_consensus.cc:3058] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:45.255893 28591 consensus_queue.cc:1035] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 02:09:45.436007 27791 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20250903 02:09:45.436285 27791 raft_consensus.cc:421] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 1 LEADER]: Emulating election...
I20250903 02:09:45.436527 27791 raft_consensus.cc:3053] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 02:09:45.436718 27791 raft_consensus.cc:738] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d9545a5d76fe4aaea54dc551a51e1967, State: Running, Role: LEADER
I20250903 02:09:45.437176 27791 consensus_queue.cc:260] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9545a5d76fe4aaea54dc551a51e1967" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41651 } } peers { permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 } }
I20250903 02:09:45.437832 27791 raft_consensus.cc:3058] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:45.442102 27791 raft_consensus.cc:695] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 2 LEADER]: Becoming Leader. State: Replica: d9545a5d76fe4aaea54dc551a51e1967, State: Running, Role: LEADER
I20250903 02:09:45.442677 27791 consensus_queue.cc:237] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9545a5d76fe4aaea54dc551a51e1967" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41651 } } peers { permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 } }
I20250903 02:09:45.443992 27791 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250903 02:09:45.447631 27791 cluster_itest_util.cc:257] Not converged past 2 yet: 2.2 1.1
I20250903 02:09:45.551435 27791 cluster_itest_util.cc:257] Not converged past 2 yet: 2.2 1.1
I20250903 02:09:45.755455 27791 cluster_itest_util.cc:257] Not converged past 2 yet: 2.2 1.1
I20250903 02:09:45.850435 28548 raft_consensus.cc:3058] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:45.855149 28548 raft_consensus.cc:1273] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04 [term 2 FOLLOWER]: Refusing update from remote peer d9545a5d76fe4aaea54dc551a51e1967: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250903 02:09:45.856431 28591 consensus_queue.cc:1035] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250903 02:09:46.060619 27791 ts_tablet_manager-itest.cc:368] Electing peer 0...
I20250903 02:09:46.060882 27791 raft_consensus.cc:421] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 2 LEADER]: Emulating election...
I20250903 02:09:46.061112 27791 raft_consensus.cc:3053] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 2 LEADER]: Stepping down as leader of term 2
I20250903 02:09:46.061337 27791 raft_consensus.cc:738] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d9545a5d76fe4aaea54dc551a51e1967, State: Running, Role: LEADER
I20250903 02:09:46.061757 27791 consensus_queue.cc:260] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9545a5d76fe4aaea54dc551a51e1967" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41651 } } peers { permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 } }
I20250903 02:09:46.062381 27791 raft_consensus.cc:3058] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:46.066351 27791 raft_consensus.cc:695] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 3 LEADER]: Becoming Leader. State: Replica: d9545a5d76fe4aaea54dc551a51e1967, State: Running, Role: LEADER
I20250903 02:09:46.066852 27791 consensus_queue.cc:237] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9545a5d76fe4aaea54dc551a51e1967" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41651 } } peers { permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 } }
I20250903 02:09:46.068157 27791 ts_tablet_manager-itest.cc:371] Waiting for servers to agree...
I20250903 02:09:46.071835 27791 cluster_itest_util.cc:257] Not converged past 3 yet: 3.3 2.2
I20250903 02:09:46.176050 27791 cluster_itest_util.cc:257] Not converged past 3 yet: 3.3 2.2
I20250903 02:09:46.380028 27791 cluster_itest_util.cc:257] Not converged past 3 yet: 3.3 2.2
I20250903 02:09:46.529544 28548 raft_consensus.cc:3058] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:46.534371 28548 raft_consensus.cc:1273] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04 [term 3 FOLLOWER]: Refusing update from remote peer d9545a5d76fe4aaea54dc551a51e1967: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250903 02:09:46.535775 28600 consensus_queue.cc:1035] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb60c8a8036c425a97155af7f345ea04" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250903 02:09:46.714181 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:09:46.736510 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:46.737174 27791 tablet_replica.cc:331] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967: stopping tablet replica
I20250903 02:09:46.737777 27791 raft_consensus.cc:2241] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 3 LEADER]: Raft consensus shutting down.
I20250903 02:09:46.738643 27791 raft_consensus.cc:2270] T d99684da8a334346838084271c83aa52 P d9545a5d76fe4aaea54dc551a51e1967 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:46.758891 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:09:46.768682 27791 tablet_server.cc:178] TabletServer@127.27.35.194:0 shutting down...
I20250903 02:09:46.787298 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:46.788005 27791 tablet_replica.cc:331] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04: stopping tablet replica
I20250903 02:09:46.788605 27791 raft_consensus.cc:2241] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250903 02:09:46.789069 27791 raft_consensus.cc:2270] T d99684da8a334346838084271c83aa52 P cb60c8a8036c425a97155af7f345ea04 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:46.808341 27791 tablet_server.cc:195] TabletServer@127.27.35.194:0 shutdown complete.
I20250903 02:09:46.816759 27791 master.cc:561] Master@127.27.35.254:43613 shutting down...
I20250903 02:09:46.834359 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:09:46.834944 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:46.835271 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7884271c7d804b7cbbdd4e76792056be: stopping tablet replica
I20250903 02:09:46.852428 27791 master.cc:583] Master@127.27.35.254:43613 shutdown complete.
[ OK ] LeadershipChangeReportingTest.TestUpdatedConsensusState (2594 ms)
[----------] 3 tests from LeadershipChangeReportingTest (10208 ms total)
[----------] 4 tests from TsTabletManagerITest
[ RUN ] TsTabletManagerITest.ReportOnReplicaHealthStatus
I20250903 02:09:46.882915 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:33607
I20250903 02:09:46.884071 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:46.889416 28612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:46.890280 28613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:46.891999 28615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:46.892275 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:46.893088 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:46.893270 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:46.893426 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865386893405 us; error 0 us; skew 500 ppm
I20250903 02:09:46.893929 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:46.896346 27791 webserver.cc:480] Webserver started at http://127.27.35.254:33197/ using document root <none> and password file <none>
I20250903 02:09:46.896823 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:46.896993 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:46.897251 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:46.898404 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "84420b93a3bd470ea9a52ee48d97220c"
format_stamp: "Formatted at 2025-09-03 02:09:46 on dist-test-slave-h78g"
I20250903 02:09:46.902884 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250903 02:09:46.906240 28620 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:46.906940 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 02:09:46.907218 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "84420b93a3bd470ea9a52ee48d97220c"
format_stamp: "Formatted at 2025-09-03 02:09:46 on dist-test-slave-h78g"
I20250903 02:09:46.907500 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:46.922979 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:46.924275 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:46.963492 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:33607
I20250903 02:09:46.963618 28671 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:33607 every 8 connection(s)
I20250903 02:09:46.967358 28672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:46.977973 28672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c: Bootstrap starting.
I20250903 02:09:46.982693 28672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:46.986990 28672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c: No bootstrap required, opened a new log
I20250903 02:09:46.989109 28672 raft_consensus.cc:357] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84420b93a3bd470ea9a52ee48d97220c" member_type: VOTER }
I20250903 02:09:46.989549 28672 raft_consensus.cc:383] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:46.989806 28672 raft_consensus.cc:738] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84420b93a3bd470ea9a52ee48d97220c, State: Initialized, Role: FOLLOWER
I20250903 02:09:46.990353 28672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84420b93a3bd470ea9a52ee48d97220c" member_type: VOTER }
I20250903 02:09:46.990795 28672 raft_consensus.cc:397] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:09:46.991026 28672 raft_consensus.cc:491] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:09:46.991300 28672 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:46.995406 28672 raft_consensus.cc:513] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84420b93a3bd470ea9a52ee48d97220c" member_type: VOTER }
I20250903 02:09:46.995952 28672 leader_election.cc:304] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84420b93a3bd470ea9a52ee48d97220c; no voters:
I20250903 02:09:46.997145 28672 leader_election.cc:290] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:09:46.997524 28675 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:09:46.998834 28675 raft_consensus.cc:695] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 1 LEADER]: Becoming Leader. State: Replica: 84420b93a3bd470ea9a52ee48d97220c, State: Running, Role: LEADER
I20250903 02:09:46.999431 28675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84420b93a3bd470ea9a52ee48d97220c" member_type: VOTER }
I20250903 02:09:47.000087 28672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:09:47.002532 28677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84420b93a3bd470ea9a52ee48d97220c. Latest consensus state: current_term: 1 leader_uuid: "84420b93a3bd470ea9a52ee48d97220c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84420b93a3bd470ea9a52ee48d97220c" member_type: VOTER } }
I20250903 02:09:47.002544 28676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84420b93a3bd470ea9a52ee48d97220c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84420b93a3bd470ea9a52ee48d97220c" member_type: VOTER } }
I20250903 02:09:47.003333 28677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:47.003415 28676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:47.009866 28680 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:09:47.016080 28680 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:09:47.017084 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:09:47.025020 28680 catalog_manager.cc:1349] Generated new cluster ID: 45b668a8fa78433d9c5fda22331f7b8a
I20250903 02:09:47.025310 28680 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:09:47.047623 28680 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:09:47.048898 28680 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:09:47.061846 28680 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c: Generated new TSK 0
I20250903 02:09:47.062500 28680 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:09:47.083300 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:47.089226 28693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:47.090766 28694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:47.094010 27791 server_base.cc:1047] running on GCE node
W20250903 02:09:47.093945 28696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:47.095090 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:47.095292 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:47.095420 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865387095409 us; error 0 us; skew 500 ppm
I20250903 02:09:47.095921 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:47.098169 27791 webserver.cc:480] Webserver started at http://127.27.35.193:42111/ using document root <none> and password file <none>
I20250903 02:09:47.098605 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:47.098757 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:47.098965 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:47.100142 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "1a2fd167a1fd4ccc978714abc97d3747"
format_stamp: "Formatted at 2025-09-03 02:09:47 on dist-test-slave-h78g"
I20250903 02:09:47.104879 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250903 02:09:47.108204 28701 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:47.109064 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250903 02:09:47.109331 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "1a2fd167a1fd4ccc978714abc97d3747"
format_stamp: "Formatted at 2025-09-03 02:09:47 on dist-test-slave-h78g"
I20250903 02:09:47.109629 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:47.148972 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:47.150126 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:47.151574 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:47.153913 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:47.154115 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:47.154345 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:47.154511 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:47.194365 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:34611
I20250903 02:09:47.194459 28763 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:34611 every 8 connection(s)
I20250903 02:09:47.198835 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:47.205802 28768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:47.208566 28769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:47.211905 27791 server_base.cc:1047] running on GCE node
W20250903 02:09:47.212697 28771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:47.213687 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:47.213917 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:47.214075 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865387214057 us; error 0 us; skew 500 ppm
I20250903 02:09:47.214625 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:47.214607 28764 heartbeater.cc:344] Connected to a master server at 127.27.35.254:33607
I20250903 02:09:47.215126 28764 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:47.216080 28764 heartbeater.cc:507] Master 127.27.35.254:33607 requested a full tablet report, sending...
I20250903 02:09:47.217792 27791 webserver.cc:480] Webserver started at http://127.27.35.194:44857/ using document root <none> and password file <none>
I20250903 02:09:47.218456 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:47.218459 28637 ts_manager.cc:194] Registered new tserver with Master: 1a2fd167a1fd4ccc978714abc97d3747 (127.27.35.193:34611)
I20250903 02:09:47.218822 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:47.219172 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:47.220721 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-1-root/instance:
uuid: "9e587150c341466e80ff4b79465e0350"
format_stamp: "Formatted at 2025-09-03 02:09:47 on dist-test-slave-h78g"
I20250903 02:09:47.221078 28637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46410
I20250903 02:09:47.225921 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250903 02:09:47.229283 28776 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:47.230037 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:09:47.230305 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-1-root
uuid: "9e587150c341466e80ff4b79465e0350"
format_stamp: "Formatted at 2025-09-03 02:09:47 on dist-test-slave-h78g"
I20250903 02:09:47.230548 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:47.253669 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:47.254869 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:47.256381 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:47.258589 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:47.258793 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:47.259038 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:47.259192 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:47.299119 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.194:42753
I20250903 02:09:47.299228 28838 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.194:42753 every 8 connection(s)
I20250903 02:09:47.303619 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:47.310223 28842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:47.312435 28843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:47.315050 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:47.315507 28839 heartbeater.cc:344] Connected to a master server at 127.27.35.254:33607
I20250903 02:09:47.315857 28839 heartbeater.cc:461] Registering TS with master...
W20250903 02:09:47.316143 28845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:47.316516 28839 heartbeater.cc:507] Master 127.27.35.254:33607 requested a full tablet report, sending...
I20250903 02:09:47.317021 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:47.317271 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:47.317489 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865387317469 us; error 0 us; skew 500 ppm
I20250903 02:09:47.318156 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:47.318524 28637 ts_manager.cc:194] Registered new tserver with Master: 9e587150c341466e80ff4b79465e0350 (127.27.35.194:42753)
I20250903 02:09:47.320039 28637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46416
I20250903 02:09:47.320647 27791 webserver.cc:480] Webserver started at http://127.27.35.195:45887/ using document root <none> and password file <none>
I20250903 02:09:47.321214 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:47.321412 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:47.321662 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:47.322836 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-2-root/instance:
uuid: "42104b62bc2c4d08970c5079187300b7"
format_stamp: "Formatted at 2025-09-03 02:09:47 on dist-test-slave-h78g"
I20250903 02:09:47.327172 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250903 02:09:47.330474 28850 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:47.331255 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 02:09:47.331503 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-2-root
uuid: "42104b62bc2c4d08970c5079187300b7"
format_stamp: "Formatted at 2025-09-03 02:09:47 on dist-test-slave-h78g"
I20250903 02:09:47.331739 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.ReportOnReplicaHealthStatus.1756865376507568-27791-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:47.343947 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:47.345110 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:47.346482 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:47.348696 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:47.348872 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:47.349108 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:47.349273 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:47.416782 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.195:41019
I20250903 02:09:47.417255 28912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.195:41019 every 8 connection(s)
I20250903 02:09:47.430589 28913 heartbeater.cc:344] Connected to a master server at 127.27.35.254:33607
I20250903 02:09:47.430931 28913 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:47.431634 28913 heartbeater.cc:507] Master 127.27.35.254:33607 requested a full tablet report, sending...
I20250903 02:09:47.433514 28637 ts_manager.cc:194] Registered new tserver with Master: 42104b62bc2c4d08970c5079187300b7 (127.27.35.195:41019)
I20250903 02:09:47.434520 27791 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013371877s
I20250903 02:09:47.435042 28637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46432
I20250903 02:09:47.460003 28637 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20250903 02:09:47.461491 28637 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 02:09:47.503398 28804 tablet_service.cc:1468] Processing CreateTablet for tablet e6b6c84d29974d9585128fed6cd11ee9 (DEFAULT_TABLE table=test-table [id=f50c025a78ca4070802a59da9430b806]), partition=
I20250903 02:09:47.504554 28804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6b6c84d29974d9585128fed6cd11ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:47.506479 28878 tablet_service.cc:1468] Processing CreateTablet for tablet e6b6c84d29974d9585128fed6cd11ee9 (DEFAULT_TABLE table=test-table [id=f50c025a78ca4070802a59da9430b806]), partition=
I20250903 02:09:47.507586 28878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6b6c84d29974d9585128fed6cd11ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:47.510282 28729 tablet_service.cc:1468] Processing CreateTablet for tablet e6b6c84d29974d9585128fed6cd11ee9 (DEFAULT_TABLE table=test-table [id=f50c025a78ca4070802a59da9430b806]), partition=
I20250903 02:09:47.511307 28729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6b6c84d29974d9585128fed6cd11ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:47.521165 28928 tablet_bootstrap.cc:492] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350: Bootstrap starting.
I20250903 02:09:47.527281 28928 tablet_bootstrap.cc:654] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:47.528414 28929 tablet_bootstrap.cc:492] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7: Bootstrap starting.
I20250903 02:09:47.529870 28930 tablet_bootstrap.cc:492] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747: Bootstrap starting.
I20250903 02:09:47.533442 28928 tablet_bootstrap.cc:492] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350: No bootstrap required, opened a new log
I20250903 02:09:47.534065 28928 ts_tablet_manager.cc:1397] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20250903 02:09:47.535223 28929 tablet_bootstrap.cc:654] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:47.537796 28928 raft_consensus.cc:357] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e587150c341466e80ff4b79465e0350" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42753 } } peers { permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 } } peers { permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 } }
I20250903 02:09:47.538882 28928 raft_consensus.cc:383] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:47.539278 28930 tablet_bootstrap.cc:654] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:47.539319 28928 raft_consensus.cc:738] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e587150c341466e80ff4b79465e0350, State: Initialized, Role: FOLLOWER
I20250903 02:09:47.540210 28928 consensus_queue.cc:260] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e587150c341466e80ff4b79465e0350" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42753 } } peers { permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 } } peers { permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 } }
I20250903 02:09:47.540949 28929 tablet_bootstrap.cc:492] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7: No bootstrap required, opened a new log
I20250903 02:09:47.541405 28929 ts_tablet_manager.cc:1397] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20250903 02:09:47.544821 28928 ts_tablet_manager.cc:1428] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20250903 02:09:47.544948 28839 heartbeater.cc:499] Master 127.27.35.254:33607 was elected leader, sending a full tablet report...
I20250903 02:09:47.546173 28929 raft_consensus.cc:357] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e587150c341466e80ff4b79465e0350" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42753 } } peers { permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 } } peers { permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 } }
I20250903 02:09:47.546988 28929 raft_consensus.cc:383] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:47.547377 28929 raft_consensus.cc:738] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42104b62bc2c4d08970c5079187300b7, State: Initialized, Role: FOLLOWER
I20250903 02:09:47.548031 28929 consensus_queue.cc:260] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e587150c341466e80ff4b79465e0350" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42753 } } peers { permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 } } peers { permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 } }
I20250903 02:09:47.548308 28930 tablet_bootstrap.cc:492] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747: No bootstrap required, opened a new log
I20250903 02:09:47.548908 28930 ts_tablet_manager.cc:1397] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.007s
I20250903 02:09:47.551666 28913 heartbeater.cc:499] Master 127.27.35.254:33607 was elected leader, sending a full tablet report...
I20250903 02:09:47.552019 28929 ts_tablet_manager.cc:1428] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20250903 02:09:47.552560 28930 raft_consensus.cc:357] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e587150c341466e80ff4b79465e0350" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42753 } } peers { permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 } } peers { permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 } }
I20250903 02:09:47.553355 28930 raft_consensus.cc:383] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:47.553648 28930 raft_consensus.cc:738] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2fd167a1fd4ccc978714abc97d3747, State: Initialized, Role: FOLLOWER
I20250903 02:09:47.554375 28930 consensus_queue.cc:260] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e587150c341466e80ff4b79465e0350" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42753 } } peers { permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 } } peers { permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 } }
I20250903 02:09:47.557921 28933 raft_consensus.cc:491] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:47.558434 28933 raft_consensus.cc:513] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e587150c341466e80ff4b79465e0350" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42753 } } peers { permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 } } peers { permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 } }
I20250903 02:09:47.562618 28933 leader_election.cc:290] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a2fd167a1fd4ccc978714abc97d3747 (127.27.35.193:34611), 42104b62bc2c4d08970c5079187300b7 (127.27.35.195:41019)
I20250903 02:09:47.574955 28764 heartbeater.cc:499] Master 127.27.35.254:33607 was elected leader, sending a full tablet report...
I20250903 02:09:47.575605 28930 ts_tablet_manager.cc:1428] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747: Time spent starting tablet: real 0.026s user 0.009s sys 0.015s
I20250903 02:09:47.579849 28739 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6b6c84d29974d9585128fed6cd11ee9" candidate_uuid: "9e587150c341466e80ff4b79465e0350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a2fd167a1fd4ccc978714abc97d3747" is_pre_election: true
I20250903 02:09:47.580765 28739 raft_consensus.cc:2466] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e587150c341466e80ff4b79465e0350 in term 0.
I20250903 02:09:47.582182 28888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6b6c84d29974d9585128fed6cd11ee9" candidate_uuid: "9e587150c341466e80ff4b79465e0350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42104b62bc2c4d08970c5079187300b7" is_pre_election: true
I20250903 02:09:47.582320 28780 leader_election.cc:304] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a2fd167a1fd4ccc978714abc97d3747, 9e587150c341466e80ff4b79465e0350; no voters:
I20250903 02:09:47.582932 28888 raft_consensus.cc:2466] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e587150c341466e80ff4b79465e0350 in term 0.
I20250903 02:09:47.583300 28933 raft_consensus.cc:2802] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 02:09:47.583616 28933 raft_consensus.cc:491] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:09:47.583979 28933 raft_consensus.cc:3058] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:47.589243 28933 raft_consensus.cc:513] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e587150c341466e80ff4b79465e0350" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42753 } } peers { permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 } } peers { permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 } }
I20250903 02:09:47.590678 28933 leader_election.cc:290] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [CANDIDATE]: Term 1 election: Requested vote from peers 1a2fd167a1fd4ccc978714abc97d3747 (127.27.35.193:34611), 42104b62bc2c4d08970c5079187300b7 (127.27.35.195:41019)
I20250903 02:09:47.591526 28739 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6b6c84d29974d9585128fed6cd11ee9" candidate_uuid: "9e587150c341466e80ff4b79465e0350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a2fd167a1fd4ccc978714abc97d3747"
I20250903 02:09:47.591661 28888 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6b6c84d29974d9585128fed6cd11ee9" candidate_uuid: "9e587150c341466e80ff4b79465e0350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42104b62bc2c4d08970c5079187300b7"
I20250903 02:09:47.592010 28739 raft_consensus.cc:3058] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:47.592191 28888 raft_consensus.cc:3058] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:47.596796 28739 raft_consensus.cc:2466] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e587150c341466e80ff4b79465e0350 in term 1.
I20250903 02:09:47.596851 28888 raft_consensus.cc:2466] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e587150c341466e80ff4b79465e0350 in term 1.
I20250903 02:09:47.597832 28780 leader_election.cc:304] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a2fd167a1fd4ccc978714abc97d3747, 9e587150c341466e80ff4b79465e0350; no voters:
I20250903 02:09:47.598513 28933 raft_consensus.cc:2802] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:09:47.599382 28933 raft_consensus.cc:695] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 1 LEADER]: Becoming Leader. State: Replica: 9e587150c341466e80ff4b79465e0350, State: Running, Role: LEADER
I20250903 02:09:47.600025 28933 consensus_queue.cc:237] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e587150c341466e80ff4b79465e0350" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42753 } } peers { permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 } } peers { permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 } }
I20250903 02:09:47.607570 28635 catalog_manager.cc:5582] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e587150c341466e80ff4b79465e0350 (127.27.35.194). New cstate: current_term: 1 leader_uuid: "9e587150c341466e80ff4b79465e0350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e587150c341466e80ff4b79465e0350" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 } health_report { overall_health: UNKNOWN } } }
I20250903 02:09:48.014132 28933 consensus_queue.cc:1035] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a2fd167a1fd4ccc978714abc97d3747" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 34611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 02:09:48.024457 28943 consensus_queue.cc:1035] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42104b62bc2c4d08970c5079187300b7" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 02:09:48.675617 27791 tablet_replica.cc:331] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747: stopping tablet replica
I20250903 02:09:48.676231 27791 raft_consensus.cc:2241] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 02:09:48.676698 27791 raft_consensus.cc:2270] T e6b6c84d29974d9585128fed6cd11ee9 P 1a2fd167a1fd4ccc978714abc97d3747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:48.987195 28780 consensus_queue.cc:579] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [LEADER]: The tablet replica hosted on peer 1a2fd167a1fd4ccc978714abc97d3747 has failed
W20250903 02:09:48.987654 28780 consensus_peers.cc:489] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 -> Peer 1a2fd167a1fd4ccc978714abc97d3747 (127.27.35.193:34611): Couldn't send request to peer 1a2fd167a1fd4ccc978714abc97d3747. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20250903 02:09:49.200268 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:09:49.219691 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:49.237035 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:09:49.245213 27791 tablet_server.cc:178] TabletServer@127.27.35.194:0 shutting down...
I20250903 02:09:49.261060 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:49.261623 27791 tablet_replica.cc:331] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350: stopping tablet replica
I20250903 02:09:49.262116 27791 raft_consensus.cc:2241] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:09:49.262845 27791 raft_consensus.cc:2270] T e6b6c84d29974d9585128fed6cd11ee9 P 9e587150c341466e80ff4b79465e0350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:49.281512 27791 tablet_server.cc:195] TabletServer@127.27.35.194:0 shutdown complete.
I20250903 02:09:49.289551 27791 tablet_server.cc:178] TabletServer@127.27.35.195:0 shutting down...
I20250903 02:09:49.306185 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:49.306774 27791 tablet_replica.cc:331] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7: stopping tablet replica
I20250903 02:09:49.307281 27791 raft_consensus.cc:2241] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 02:09:49.307703 27791 raft_consensus.cc:2270] T e6b6c84d29974d9585128fed6cd11ee9 P 42104b62bc2c4d08970c5079187300b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:49.325434 27791 tablet_server.cc:195] TabletServer@127.27.35.195:0 shutdown complete.
I20250903 02:09:49.333029 27791 master.cc:561] Master@127.27.35.254:33607 shutting down...
I20250903 02:09:49.345722 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:09:49.346216 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:49.346510 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P 84420b93a3bd470ea9a52ee48d97220c: stopping tablet replica
I20250903 02:09:49.363582 27791 master.cc:583] Master@127.27.35.254:33607 shutdown complete.
[ OK ] TsTabletManagerITest.ReportOnReplicaHealthStatus (2504 ms)
[ RUN ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters
I20250903 02:09:49.388533 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:40629
I20250903 02:09:49.389642 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:49.394752 28952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:49.396212 28953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:49.397735 28955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:49.398876 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:49.399564 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:49.399878 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:49.400034 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865389400009 us; error 0 us; skew 500 ppm
I20250903 02:09:49.400489 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:49.402732 27791 webserver.cc:480] Webserver started at http://127.27.35.254:44527/ using document root <none> and password file <none>
I20250903 02:09:49.403172 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:49.403321 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:49.403524 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:49.404584 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "0374f2a950a641ba82c390fff8200e97"
format_stamp: "Formatted at 2025-09-03 02:09:49 on dist-test-slave-h78g"
I20250903 02:09:49.408914 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:09:49.412366 28960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:49.413084 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 02:09:49.413393 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "0374f2a950a641ba82c390fff8200e97"
format_stamp: "Formatted at 2025-09-03 02:09:49 on dist-test-slave-h78g"
I20250903 02:09:49.413661 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:49.441582 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:49.442595 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:49.478832 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:40629
I20250903 02:09:49.478927 29011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:40629 every 8 connection(s)
I20250903 02:09:49.482724 29012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:49.493328 29012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97: Bootstrap starting.
I20250903 02:09:49.497567 29012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:49.501655 29012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97: No bootstrap required, opened a new log
I20250903 02:09:49.503924 29012 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0374f2a950a641ba82c390fff8200e97" member_type: VOTER }
I20250903 02:09:49.504325 29012 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:49.504510 29012 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374f2a950a641ba82c390fff8200e97, State: Initialized, Role: FOLLOWER
I20250903 02:09:49.505013 29012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0374f2a950a641ba82c390fff8200e97" member_type: VOTER }
I20250903 02:09:49.505481 29012 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:09:49.505667 29012 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:09:49.505882 29012 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:49.510653 29012 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0374f2a950a641ba82c390fff8200e97" member_type: VOTER }
I20250903 02:09:49.511379 29012 leader_election.cc:304] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0374f2a950a641ba82c390fff8200e97; no voters:
I20250903 02:09:49.512866 29012 leader_election.cc:290] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:09:49.513170 29015 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:09:49.514760 29015 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 1 LEADER]: Becoming Leader. State: Replica: 0374f2a950a641ba82c390fff8200e97, State: Running, Role: LEADER
I20250903 02:09:49.515441 29015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0374f2a950a641ba82c390fff8200e97" member_type: VOTER }
I20250903 02:09:49.516201 29012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:09:49.518509 29016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0374f2a950a641ba82c390fff8200e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0374f2a950a641ba82c390fff8200e97" member_type: VOTER } }
I20250903 02:09:49.519379 29016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:49.518615 29017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0374f2a950a641ba82c390fff8200e97. Latest consensus state: current_term: 1 leader_uuid: "0374f2a950a641ba82c390fff8200e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0374f2a950a641ba82c390fff8200e97" member_type: VOTER } }
I20250903 02:09:49.520896 29017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:49.524638 29021 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:09:49.532647 29021 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:09:49.533658 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:09:49.541790 29021 catalog_manager.cc:1349] Generated new cluster ID: a7988d1b888f4a0dbec6f7425b2b8a45
I20250903 02:09:49.542125 29021 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:09:49.556452 29021 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:09:49.557703 29021 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:09:49.569720 29021 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97: Generated new TSK 0
I20250903 02:09:49.570348 29021 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:09:49.600867 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:49.606714 29033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:49.607950 29034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:49.609787 29036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:49.609920 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:49.610813 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:49.610998 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:49.611177 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865389611162 us; error 0 us; skew 500 ppm
I20250903 02:09:49.611713 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:49.614141 27791 webserver.cc:480] Webserver started at http://127.27.35.193:38429/ using document root <none> and password file <none>
I20250903 02:09:49.614627 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:49.614791 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:49.615083 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:49.616321 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "8cef9d5ecb054ca197751ef297a9cac9"
format_stamp: "Formatted at 2025-09-03 02:09:49 on dist-test-slave-h78g"
I20250903 02:09:49.621012 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:09:49.624378 29041 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:49.625119 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 02:09:49.625416 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "8cef9d5ecb054ca197751ef297a9cac9"
format_stamp: "Formatted at 2025-09-03 02:09:49 on dist-test-slave-h78g"
I20250903 02:09:49.625684 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:49.646216 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:49.647323 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:49.648801 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:49.651046 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:49.651255 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:49.651496 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:49.651647 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:49.691272 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:39959
I20250903 02:09:49.691378 29103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:39959 every 8 connection(s)
I20250903 02:09:49.709976 29104 heartbeater.cc:344] Connected to a master server at 127.27.35.254:40629
I20250903 02:09:49.710337 29104 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:49.710975 29104 heartbeater.cc:507] Master 127.27.35.254:40629 requested a full tablet report, sending...
I20250903 02:09:49.713109 28977 ts_manager.cc:194] Registered new tserver with Master: 8cef9d5ecb054ca197751ef297a9cac9 (127.27.35.193:39959)
I20250903 02:09:49.713289 27791 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.018482554s
I20250903 02:09:49.715663 28977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44234
I20250903 02:09:49.735136 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:09:49.751340 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:49.766742 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:09:49.774320 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 02:09:49.777007 27791 tablet_server.cc:93] deduplicated master addresses: 127.27.35.254:40629
W20250903 02:09:49.780412 29114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:49.781258 29115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:49.782052 29117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:49.782862 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:49.783615 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:49.783842 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:49.783974 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865389783962 us; error 0 us; skew 500 ppm
I20250903 02:09:49.784430 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:49.786710 27791 webserver.cc:480] Webserver started at http://127.27.35.193:38429/ using document root <none> and password file <none>
I20250903 02:09:49.787149 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:49.787288 27791 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250903 02:09:49.790684 27791 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20250903 02:09:49.793292 29122 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:49.794045 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 02:09:49.794339 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "8cef9d5ecb054ca197751ef297a9cac9"
format_stamp: "Formatted at 2025-09-03 02:09:49 on dist-test-slave-h78g"
I20250903 02:09:49.794611 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:49.814307 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:49.815297 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:49.816711 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:49.819181 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:49.819376 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:49.819594 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:49.819739 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:49.858561 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:39959
I20250903 02:09:49.858669 29184 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:39959 every 8 connection(s)
I20250903 02:09:49.864152 27791 tablet_server.cc:178] TabletServer@127.27.35.193:39959 shutting down...
I20250903 02:09:49.882326 29185 heartbeater.cc:344] Connected to a master server at 127.27.35.254:40629
I20250903 02:09:49.883005 29185 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:49.883880 29185 heartbeater.cc:507] Master 127.27.35.254:40629 requested a full tablet report, sending...
I20250903 02:09:49.886123 28977 ts_manager.cc:194] Re-registered known tserver with Master: 8cef9d5ecb054ca197751ef297a9cac9 (127.27.35.193:39959)
I20250903 02:09:49.887447 28977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44258
I20250903 02:09:49.889698 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:09:49.904927 27791 tablet_server.cc:195] TabletServer@127.27.35.193:39959 shutdown complete.
I20250903 02:09:49.911976 27791 master.cc:561] Master@127.27.35.254:40629 shutting down...
I20250903 02:09:49.925154 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:09:49.925638 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:49.926033 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P 0374f2a950a641ba82c390fff8200e97: stopping tablet replica
I20250903 02:09:49.944376 27791 master.cc:583] Master@127.27.35.254:40629 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeduplicateMasterAddrsForHeartbeaters (578 ms)
[ RUN ] TsTabletManagerITest.TestTableStats
I20250903 02:09:49.966329 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:41877,127.27.35.253:42649,127.27.35.252:45587
I20250903 02:09:49.967555 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:49.971982 29192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:49.973399 29193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:49.974643 29195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:49.975431 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:49.976297 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:49.976466 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:49.976636 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865389976608 us; error 0 us; skew 500 ppm
I20250903 02:09:49.977152 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:49.979359 27791 webserver.cc:480] Webserver started at http://127.27.35.254:38427/ using document root <none> and password file <none>
I20250903 02:09:49.979837 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:49.980019 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:49.980289 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:49.981321 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "6e1ec9fa14084d698da867035b48fc13"
format_stamp: "Formatted at 2025-09-03 02:09:49 on dist-test-slave-h78g"
I20250903 02:09:49.985780 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:09:49.989063 29200 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:49.989861 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:09:49.990111 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "6e1ec9fa14084d698da867035b48fc13"
format_stamp: "Formatted at 2025-09-03 02:09:49 on dist-test-slave-h78g"
I20250903 02:09:49.990366 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:50.008109 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:50.009063 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:50.044093 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:41877
I20250903 02:09:50.044162 29251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:41877 every 8 connection(s)
I20250903 02:09:50.047696 29252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:50.048043 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:50.053663 29254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:50.054627 29255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.054692 29252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
I20250903 02:09:50.057718 27791 server_base.cc:1047] running on GCE node
W20250903 02:09:50.058975 29257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.059881 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:50.060150 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:50.060344 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865390060325 us; error 0 us; skew 500 ppm
I20250903 02:09:50.061054 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:50.063573 27791 webserver.cc:480] Webserver started at http://127.27.35.253:36125/ using document root <none> and password file <none>
I20250903 02:09:50.064203 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:50.064462 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:50.064769 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:50.066110 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-1-root/instance:
uuid: "36f57bf97db44ab1a86d32e59b4ad438"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:50.068732 29252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } has no permanent_uuid. Determining permanent_uuid...
I20250903 02:09:50.070940 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
W20250903 02:09:50.070964 29201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.35.253:42649: connect: Connection refused (error 111)
W20250903 02:09:50.074196 29252 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.27.35.253:42649: Network error: Client connection negotiation failed: client connection to 127.27.35.253:42649: connect: Connection refused (error 111)
I20250903 02:09:50.075201 29265 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.076037 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:09:50.076318 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-1-root
uuid: "36f57bf97db44ab1a86d32e59b4ad438"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:50.076604 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:50.099125 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:50.100376 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:50.125734 29252 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } attempt: 1
W20250903 02:09:50.130586 29252 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.27.35.253:42649: Network error: Client connection negotiation failed: client connection to 127.27.35.253:42649: connect: Connection refused (error 111)
I20250903 02:09:50.138574 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.253:42649
I20250903 02:09:50.138695 29316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.253:42649 every 8 connection(s)
I20250903 02:09:50.142333 29317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:50.142617 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:50.148168 29319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:50.148716 29320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.149924 29317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
I20250903 02:09:50.151248 27791 server_base.cc:1047] running on GCE node
W20250903 02:09:50.152328 29323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.153167 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:50.153399 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:50.153564 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865390153546 us; error 0 us; skew 500 ppm
I20250903 02:09:50.154076 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:50.156782 27791 webserver.cc:480] Webserver started at http://127.27.35.252:41231/ using document root <none> and password file <none>
I20250903 02:09:50.157294 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:50.157490 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:50.157729 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:50.159044 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-2-root/instance:
uuid: "6f21cd83f94b4a75ac8648f471aeb85b"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:50.161785 29317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } has no permanent_uuid. Determining permanent_uuid...
I20250903 02:09:50.165864 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250903 02:09:50.169667 29329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.170614 27791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250903 02:09:50.170956 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-2-root
uuid: "6f21cd83f94b4a75ac8648f471aeb85b"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:50.171272 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:50.173748 29317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
W20250903 02:09:50.179939 29317 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.27.35.252:45587: Network error: Client connection negotiation failed: client connection to 127.27.35.252:45587: connect: Connection refused (error 111)
I20250903 02:09:50.180814 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:50.181814 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:50.192214 29252 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } attempt: 2
I20250903 02:09:50.198510 29317 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } attempt: 1
W20250903 02:09:50.204324 29317 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.27.35.252:45587: Network error: Client connection negotiation failed: client connection to 127.27.35.252:45587: connect: Connection refused (error 111)
I20250903 02:09:50.206319 29252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
W20250903 02:09:50.211313 29252 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.27.35.252:45587: Network error: Client connection negotiation failed: client connection to 127.27.35.252:45587: connect: Connection refused (error 111)
I20250903 02:09:50.226918 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.252:45587
I20250903 02:09:50.227018 29381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.252:45587 every 8 connection(s)
I20250903 02:09:50.230047 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:09:50.231177 29382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:50.236035 29252 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } attempt: 1
I20250903 02:09:50.238108 29382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
I20250903 02:09:50.250902 29382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } has no permanent_uuid. Determining permanent_uuid...
I20250903 02:09:50.254138 29252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13: Bootstrap starting.
I20250903 02:09:50.258424 29252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:50.258838 29317 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } attempt: 2
I20250903 02:09:50.261651 29382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
I20250903 02:09:50.263576 29252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13: No bootstrap required, opened a new log
I20250903 02:09:50.266505 29252 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:50.267270 29252 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:50.267577 29252 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1ec9fa14084d698da867035b48fc13, State: Initialized, Role: FOLLOWER
I20250903 02:09:50.268311 29252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:50.270519 29388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:50.271907 29388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:50.272373 29252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:09:50.278170 29317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438: Bootstrap starting.
I20250903 02:09:50.283166 29382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: Bootstrap starting.
I20250903 02:09:50.285646 29317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:50.288058 29382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:50.290387 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250903 02:09:50.291810 29317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438: No bootstrap required, opened a new log
I20250903 02:09:50.293275 29382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: No bootstrap required, opened a new log
I20250903 02:09:50.293949 29317 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:50.294477 29317 raft_consensus.cc:383] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:50.294660 29317 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f57bf97db44ab1a86d32e59b4ad438, State: Initialized, Role: FOLLOWER
I20250903 02:09:50.295166 29317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:50.295902 29382 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
W20250903 02:09:50.296285 29400 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 02:09:50.296794 29400 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 02:09:50.296725 29382 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:50.297279 29382 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f21cd83f94b4a75ac8648f471aeb85b, State: Initialized, Role: FOLLOWER
I20250903 02:09:50.297000 29402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:50.297830 29402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:50.297950 29382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:50.298646 29317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:09:50.299782 29388 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:50.299903 29403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:50.300275 29388 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:50.300875 29403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:50.301735 29382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:09:50.302860 29388 leader_election.cc:290] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649), 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587)
I20250903 02:09:50.303126 29403 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:50.303582 29403 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:50.305549 29403 leader_election.cc:290] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649)
I20250903 02:09:50.305781 29357 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e1ec9fa14084d698da867035b48fc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" is_pre_election: true
I20250903 02:09:50.305891 29292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e1ec9fa14084d698da867035b48fc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f57bf97db44ab1a86d32e59b4ad438" is_pre_election: true
I20250903 02:09:50.306617 29357 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1ec9fa14084d698da867035b48fc13 in term 0.
I20250903 02:09:50.306870 29292 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1ec9fa14084d698da867035b48fc13 in term 0.
I20250903 02:09:50.307560 29291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f57bf97db44ab1a86d32e59b4ad438" is_pre_election: true
I20250903 02:09:50.308096 29291 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f21cd83f94b4a75ac8648f471aeb85b in term 0.
I20250903 02:09:50.309085 29201 leader_election.cc:304] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438, 6e1ec9fa14084d698da867035b48fc13; no voters:
I20250903 02:09:50.309085 29330 leader_election.cc:304] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438, 6f21cd83f94b4a75ac8648f471aeb85b; no voters:
I20250903 02:09:50.310652 29388 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 02:09:50.310786 29227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1ec9fa14084d698da867035b48fc13" is_pre_election: true
I20250903 02:09:50.311004 29388 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:09:50.311533 29388 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:50.316051 29403 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 02:09:50.316613 29403 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:09:50.317121 29403 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:50.317652 29388 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:50.318941 29227 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6f21cd83f94b4a75ac8648f471aeb85b in current term 1: Already voted for candidate 6e1ec9fa14084d698da867035b48fc13 in this term.
I20250903 02:09:50.324502 29388 leader_election.cc:290] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [CANDIDATE]: Term 1 election: Requested vote from peers 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649), 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587)
I20250903 02:09:50.325672 29291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e1ec9fa14084d698da867035b48fc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f57bf97db44ab1a86d32e59b4ad438"
I20250903 02:09:50.325824 29357 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e1ec9fa14084d698da867035b48fc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f21cd83f94b4a75ac8648f471aeb85b"
I20250903 02:09:50.326287 29291 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:50.328341 29403 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:50.329495 29357 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e1ec9fa14084d698da867035b48fc13 in current term 1: Already voted for candidate 6f21cd83f94b4a75ac8648f471aeb85b in this term.
I20250903 02:09:50.332214 29403 leader_election.cc:290] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 1 election: Requested vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649)
I20250903 02:09:50.333278 29292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f57bf97db44ab1a86d32e59b4ad438"
I20250903 02:09:50.333406 29227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1ec9fa14084d698da867035b48fc13"
I20250903 02:09:50.335064 29227 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6f21cd83f94b4a75ac8648f471aeb85b in current term 1: Already voted for candidate 6e1ec9fa14084d698da867035b48fc13 in this term.
I20250903 02:09:50.335906 29291 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1ec9fa14084d698da867035b48fc13 in term 1.
I20250903 02:09:50.336632 29330 leader_election.cc:304] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f21cd83f94b4a75ac8648f471aeb85b; no voters: 36f57bf97db44ab1a86d32e59b4ad438, 6e1ec9fa14084d698da867035b48fc13
I20250903 02:09:50.337882 29403 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250903 02:09:50.338862 29201 leader_election.cc:304] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438, 6e1ec9fa14084d698da867035b48fc13; no voters: 6f21cd83f94b4a75ac8648f471aeb85b
I20250903 02:09:50.339538 29388 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:09:50.341625 29388 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 1 LEADER]: Becoming Leader. State: Replica: 6e1ec9fa14084d698da867035b48fc13, State: Running, Role: LEADER
I20250903 02:09:50.342532 29388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:50.343152 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250903 02:09:50.347697 29420 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250903 02:09:50.347321 29418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e1ec9fa14084d698da867035b48fc13. Latest consensus state: current_term: 1 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
W20250903 02:09:50.348130 29420 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 02:09:50.348201 29418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:50.350328 29423 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:09:50.354967 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:50.356487 29428 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250903 02:09:50.356779 29428 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250903 02:09:50.357694 29423 catalog_manager.cc:1486] Initializing Kudu cluster ID...
W20250903 02:09:50.363484 29429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:50.364427 29430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:50.370741 29432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.370792 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:50.372119 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:50.372408 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:50.372642 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865390372618 us; error 0 us; skew 500 ppm
I20250903 02:09:50.373358 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:50.373904 29357 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 1 FOLLOWER]: Refusing update from remote peer 6e1ec9fa14084d698da867035b48fc13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 02:09:50.373979 29291 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 1 FOLLOWER]: Refusing update from remote peer 6e1ec9fa14084d698da867035b48fc13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 02:09:50.375489 29388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250903 02:09:50.376472 27791 webserver.cc:480] Webserver started at http://127.27.35.193:36477/ using document root <none> and password file <none>
I20250903 02:09:50.376422 29418 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 02:09:50.377084 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:50.377357 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:50.377691 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:50.378999 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "9c933d23ae9746a8858c7d3ed33c3129"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:50.384311 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250903 02:09:50.388121 29440 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.389163 27791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250903 02:09:50.389544 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "9c933d23ae9746a8858c7d3ed33c3129"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:50.389901 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:50.401033 29402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e1ec9fa14084d698da867035b48fc13. Latest consensus state: current_term: 1 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:50.401731 29402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:50.401727 29403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e1ec9fa14084d698da867035b48fc13. Latest consensus state: current_term: 1 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:50.402650 29403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:50.407507 29447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:50.407668 29418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:50.408385 29447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:50.408495 29418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:50.408911 29437 mvcc.cc:204] Tried to move back new op lower bound from 7196120638957641728 to 7196120638853160960. Current Snapshot: MvccSnapshot[applied={T|T < 7196120638957641728}]
I20250903 02:09:50.411092 29402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:50.412027 29402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:50.412672 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:50.414245 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:50.414783 29423 catalog_manager.cc:1349] Generated new cluster ID: 7eaaf77333364ef996cd27c71b6738f8
I20250903 02:09:50.415087 29423 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:09:50.415016 29403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:50.415776 29403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:50.416445 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:50.419826 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:50.420126 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.420424 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:50.420634 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.451567 29423 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:09:50.453296 29423 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:09:50.480888 29423 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13: Generated new TSK 0
I20250903 02:09:50.481597 29423 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:09:50.493815 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:41955
I20250903 02:09:50.494068 29508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:41955 every 8 connection(s)
I20250903 02:09:50.500501 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 02:09:50.516147 29510 heartbeater.cc:344] Connected to a master server at 127.27.35.253:42649
I20250903 02:09:50.516575 29510 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:50.517499 29510 heartbeater.cc:507] Master 127.27.35.253:42649 requested a full tablet report, sending...
W20250903 02:09:50.518659 29516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.519973 29509 heartbeater.cc:344] Connected to a master server at 127.27.35.252:45587
I20250903 02:09:50.520431 29509 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:50.521384 29282 ts_manager.cc:194] Registered new tserver with Master: 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:09:50.521427 29509 heartbeater.cc:507] Master 127.27.35.252:45587 requested a full tablet report, sending...
I20250903 02:09:50.526594 29347 ts_manager.cc:194] Registered new tserver with Master: 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
W20250903 02:09:50.526973 29517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.528829 29511 heartbeater.cc:344] Connected to a master server at 127.27.35.254:41877
I20250903 02:09:50.529309 29511 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:50.530419 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:50.530349 29511 heartbeater.cc:507] Master 127.27.35.254:41877 requested a full tablet report, sending...
W20250903 02:09:50.530647 29519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.531914 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:50.532191 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:50.532410 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865390532391 us; error 0 us; skew 500 ppm
I20250903 02:09:50.532930 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:50.533066 29217 ts_manager.cc:194] Registered new tserver with Master: 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:09:50.535013 29217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38906
I20250903 02:09:50.535456 27791 webserver.cc:480] Webserver started at http://127.27.35.194:38475/ using document root <none> and password file <none>
I20250903 02:09:50.536041 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:50.536270 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:50.536528 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:50.537537 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-1-root/instance:
uuid: "7b012cbd73294b4a869a1515bfd63348"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:50.541812 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:09:50.545145 29524 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.545929 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250903 02:09:50.546218 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-1-root
uuid: "7b012cbd73294b4a869a1515bfd63348"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:50.546504 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:50.568745 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:50.569921 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:50.571900 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:50.575400 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:50.575615 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.575876 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:50.576035 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.637390 29511 heartbeater.cc:499] Master 127.27.35.254:41877 was elected leader, sending a full tablet report...
I20250903 02:09:50.654629 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.194:42673
I20250903 02:09:50.654713 29586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.194:42673 every 8 connection(s)
I20250903 02:09:50.664523 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 02:09:50.694311 29588 heartbeater.cc:344] Connected to a master server at 127.27.35.253:42649
I20250903 02:09:50.695237 29587 heartbeater.cc:344] Connected to a master server at 127.27.35.252:45587
I20250903 02:09:50.695888 29588 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:50.697751 29587 heartbeater.cc:461] Registering TS with master...
W20250903 02:09:50.698098 29594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.698570 29588 heartbeater.cc:507] Master 127.27.35.253:42649 requested a full tablet report, sending...
I20250903 02:09:50.703510 29587 heartbeater.cc:507] Master 127.27.35.252:45587 requested a full tablet report, sending...
I20250903 02:09:50.706558 29282 ts_manager.cc:194] Registered new tserver with Master: 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:09:50.722802 29595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.724269 29589 heartbeater.cc:344] Connected to a master server at 127.27.35.254:41877
I20250903 02:09:50.724725 29589 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:50.724994 29347 ts_manager.cc:194] Registered new tserver with Master: 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:09:50.725390 29597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:50.725652 29589 heartbeater.cc:507] Master 127.27.35.254:41877 requested a full tablet report, sending...
I20250903 02:09:50.727782 29217 ts_manager.cc:194] Registered new tserver with Master: 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:09:50.728268 27791 server_base.cc:1047] running on GCE node
I20250903 02:09:50.729357 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:50.729624 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:50.729566 29217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38910
I20250903 02:09:50.729961 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865390729941 us; error 0 us; skew 500 ppm
I20250903 02:09:50.730618 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:50.733880 27791 webserver.cc:480] Webserver started at http://127.27.35.195:32939/ using document root <none> and password file <none>
I20250903 02:09:50.734455 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:50.734687 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:09:50.734936 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:09:50.736023 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-2-root/instance:
uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:50.740345 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20250903 02:09:50.743973 29602 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.744771 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 02:09:50.745121 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-2-root
uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:50.745416 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:09:50.762737 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:50.764062 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:50.765216 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:09:50.767997 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:09:50.768213 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.768436 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:09:50.768595 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:50.833637 29589 heartbeater.cc:499] Master 127.27.35.254:41877 was elected leader, sending a full tablet report...
I20250903 02:09:50.845157 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.195:45481
I20250903 02:09:50.845260 29664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.195:45481 every 8 connection(s)
I20250903 02:09:50.864926 29665 heartbeater.cc:344] Connected to a master server at 127.27.35.252:45587
I20250903 02:09:50.865371 29665 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:50.866410 29665 heartbeater.cc:507] Master 127.27.35.252:45587 requested a full tablet report, sending...
I20250903 02:09:50.866734 29666 heartbeater.cc:344] Connected to a master server at 127.27.35.253:42649
I20250903 02:09:50.867165 29666 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:50.868093 29666 heartbeater.cc:507] Master 127.27.35.253:42649 requested a full tablet report, sending...
I20250903 02:09:50.870525 29282 ts_manager.cc:194] Registered new tserver with Master: 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:50.871109 29667 heartbeater.cc:344] Connected to a master server at 127.27.35.254:41877
I20250903 02:09:50.871121 29347 ts_manager.cc:194] Registered new tserver with Master: 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:50.871701 29667 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:50.872633 29667 heartbeater.cc:507] Master 127.27.35.254:41877 requested a full tablet report, sending...
I20250903 02:09:50.874609 29217 ts_manager.cc:194] Registered new tserver with Master: 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:50.875483 27791 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.022822826s
I20250903 02:09:50.876411 29217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38920
I20250903 02:09:50.917861 29217 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20250903 02:09:50.919629 29217 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 02:09:50.980302 29667 heartbeater.cc:499] Master 127.27.35.254:41877 was elected leader, sending a full tablet report...
I20250903 02:09:50.985947 29630 tablet_service.cc:1468] Processing CreateTablet for tablet db760726eb304ef0b69159db4bbaef75 (DEFAULT_TABLE table=test-table [id=9d7b841db0334b5897236fd155f908b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:50.986739 29629 tablet_service.cc:1468] Processing CreateTablet for tablet beaf5654ab144e23a627269917ad5ee9 (DEFAULT_TABLE table=test-table [id=9d7b841db0334b5897236fd155f908b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:50.987461 29630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db760726eb304ef0b69159db4bbaef75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:50.988157 29629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beaf5654ab144e23a627269917ad5ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:50.993034 29628 tablet_service.cc:1468] Processing CreateTablet for tablet 2da3487154804c71a24242048c1b0c16 (DEFAULT_TABLE table=test-table [id=9d7b841db0334b5897236fd155f908b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:50.994892 29628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da3487154804c71a24242048c1b0c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:51.006950 29552 tablet_service.cc:1468] Processing CreateTablet for tablet db760726eb304ef0b69159db4bbaef75 (DEFAULT_TABLE table=test-table [id=9d7b841db0334b5897236fd155f908b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:51.008634 29552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db760726eb304ef0b69159db4bbaef75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:51.007169 29551 tablet_service.cc:1468] Processing CreateTablet for tablet beaf5654ab144e23a627269917ad5ee9 (DEFAULT_TABLE table=test-table [id=9d7b841db0334b5897236fd155f908b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:51.008839 29550 tablet_service.cc:1468] Processing CreateTablet for tablet 2da3487154804c71a24242048c1b0c16 (DEFAULT_TABLE table=test-table [id=9d7b841db0334b5897236fd155f908b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:51.010255 29551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beaf5654ab144e23a627269917ad5ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:51.033783 29550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da3487154804c71a24242048c1b0c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:51.070704 29686 tablet_bootstrap.cc:492] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6: Bootstrap starting.
I20250903 02:09:51.089871 29686 tablet_bootstrap.cc:654] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:51.093364 29687 tablet_bootstrap.cc:492] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348: Bootstrap starting.
I20250903 02:09:51.100293 29687 tablet_bootstrap.cc:654] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:51.100122 29474 tablet_service.cc:1468] Processing CreateTablet for tablet db760726eb304ef0b69159db4bbaef75 (DEFAULT_TABLE table=test-table [id=9d7b841db0334b5897236fd155f908b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:51.101536 29474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db760726eb304ef0b69159db4bbaef75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:51.103454 29686 tablet_bootstrap.cc:492] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6: No bootstrap required, opened a new log
I20250903 02:09:51.105226 29686 ts_tablet_manager.cc:1397] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.002s
I20250903 02:09:51.115360 29472 tablet_service.cc:1468] Processing CreateTablet for tablet 2da3487154804c71a24242048c1b0c16 (DEFAULT_TABLE table=test-table [id=9d7b841db0334b5897236fd155f908b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:51.111137 29473 tablet_service.cc:1468] Processing CreateTablet for tablet beaf5654ab144e23a627269917ad5ee9 (DEFAULT_TABLE table=test-table [id=9d7b841db0334b5897236fd155f908b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250903 02:09:51.116361 29686 raft_consensus.cc:357] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.116902 29472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da3487154804c71a24242048c1b0c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:51.117123 29686 raft_consensus.cc:383] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:51.117578 29473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet beaf5654ab144e23a627269917ad5ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:09:51.119241 29687 tablet_bootstrap.cc:492] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348: No bootstrap required, opened a new log
I20250903 02:09:51.119871 29687 ts_tablet_manager.cc:1397] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.005s
I20250903 02:09:51.122787 29687 raft_consensus.cc:357] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.123639 29687 raft_consensus.cc:383] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:51.124023 29687 raft_consensus.cc:738] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Initialized, Role: FOLLOWER
I20250903 02:09:51.117507 29686 raft_consensus.cc:738] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cecbbd1da7046a19f694aa1fe677ed6, State: Initialized, Role: FOLLOWER
I20250903 02:09:51.125974 29687 consensus_queue.cc:260] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.135320 29690 tablet_bootstrap.cc:492] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129: Bootstrap starting.
I20250903 02:09:51.142833 29690 tablet_bootstrap.cc:654] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:51.149504 29694 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:51.150156 29694 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.153532 29686 consensus_queue.cc:260] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.135324 29687 ts_tablet_manager.cc:1428] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348: Time spent starting tablet: real 0.015s user 0.005s sys 0.003s
I20250903 02:09:51.157029 29694 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:51.166314 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:51.167001 29687 tablet_bootstrap.cc:492] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348: Bootstrap starting.
W20250903 02:09:51.168009 29525 leader_election.cc:343] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Illegal state: must be running to vote when last-logged opid is not known
I20250903 02:09:51.168298 29691 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:51.168959 29691 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.173137 29640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:51.173810 29687 tablet_bootstrap.cc:654] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:51.173822 29640 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 0.
I20250903 02:09:51.175673 29527 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348; no voters: 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:09:51.177517 29690 tablet_bootstrap.cc:492] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129: No bootstrap required, opened a new log
I20250903 02:09:51.178305 29690 ts_tablet_manager.cc:1397] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.010s
I20250903 02:09:51.185628 29687 tablet_bootstrap.cc:492] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348: No bootstrap required, opened a new log
I20250903 02:09:51.186162 29687 ts_tablet_manager.cc:1397] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20250903 02:09:51.188482 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:51.188936 29694 raft_consensus.cc:2802] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 02:09:51.189345 29694 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:09:51.189675 29694 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Advancing to term 1
W20250903 02:09:51.190470 29603 leader_election.cc:343] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Illegal state: must be running to vote when last-logged opid is not known
I20250903 02:09:51.192416 29687 raft_consensus.cc:357] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.193692 29687 raft_consensus.cc:383] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:51.194015 29687 raft_consensus.cc:738] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Initialized, Role: FOLLOWER
I20250903 02:09:51.194782 29687 consensus_queue.cc:260] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.195586 29686 ts_tablet_manager.cc:1428] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6: Time spent starting tablet: real 0.084s user 0.003s sys 0.012s
I20250903 02:09:51.196888 29694 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.198789 29694 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 election: Requested vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:51.199365 29690 raft_consensus.cc:357] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.200323 29690 raft_consensus.cc:383] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:51.200619 29690 raft_consensus.cc:738] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c933d23ae9746a8858c7d3ed33c3129, State: Initialized, Role: FOLLOWER
I20250903 02:09:51.201437 29690 consensus_queue.cc:260] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.202874 29640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:09:51.203480 29640 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:51.204773 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:09:51.210204 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
W20250903 02:09:51.210875 29525 leader_election.cc:343] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Illegal state: must be running to vote when last-logged opid is not known
I20250903 02:09:51.210899 29687 ts_tablet_manager.cc:1428] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348: Time spent starting tablet: real 0.024s user 0.007s sys 0.001s
I20250903 02:09:51.211170 29562 raft_consensus.cc:2391] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cecbbd1da7046a19f694aa1fe677ed6 in current term 1: Already voted for candidate 7b012cbd73294b4a869a1515bfd63348 in this term.
I20250903 02:09:51.212730 29604 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:09:51.215503 29687 tablet_bootstrap.cc:492] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348: Bootstrap starting.
I20250903 02:09:51.217979 29690 ts_tablet_manager.cc:1428] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129: Time spent starting tablet: real 0.039s user 0.010s sys 0.002s
I20250903 02:09:51.221138 29691 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:09:51.221109 29686 tablet_bootstrap.cc:492] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6: Bootstrap starting.
I20250903 02:09:51.224395 29694 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:51.224918 29694 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.227283 29694 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:09:51.221077 29640 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 1.
I20250903 02:09:51.228081 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:51.228798 29484 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 0.
I20250903 02:09:51.229463 29687 tablet_bootstrap.cc:654] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:51.230053 29695 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 02:09:51.232532 29686 tablet_bootstrap.cc:654] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:51.236757 29525 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:09:51.238037 29692 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 02:09:51.238494 29692 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:09:51.238845 29692 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:51.246265 29692 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.249248 29640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:51.249399 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:09:51.250041 29484 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:51.251219 29692 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 election: Requested vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:09:51.254326 29527 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348; no voters: 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:09:51.255720 29687 tablet_bootstrap.cc:492] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348: No bootstrap required, opened a new log
I20250903 02:09:51.256263 29687 ts_tablet_manager.cc:1397] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.006s
I20250903 02:09:51.257894 29484 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 1.
I20250903 02:09:51.258986 29525 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
W20250903 02:09:51.259641 29527 leader_election.cc:343] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481): Illegal state: must be running to vote when last-logged opid is not known
I20250903 02:09:51.259917 29692 raft_consensus.cc:2802] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:09:51.261608 29640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:09:51.262547 29694 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Leader election won for term 1
W20250903 02:09:51.262952 29527 leader_election.cc:343] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481): Illegal state: must be running to vote when last-logged opid is not known
I20250903 02:09:51.261255 29687 raft_consensus.cc:357] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.265560 29687 raft_consensus.cc:383] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:51.265887 29687 raft_consensus.cc:738] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Initialized, Role: FOLLOWER
I20250903 02:09:51.267614 29687 consensus_queue.cc:260] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.268599 29690 tablet_bootstrap.cc:492] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129: Bootstrap starting.
I20250903 02:09:51.267392 29686 tablet_bootstrap.cc:492] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6: No bootstrap required, opened a new log
I20250903 02:09:51.271034 29686 ts_tablet_manager.cc:1397] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6: Time spent bootstrapping tablet: real 0.050s user 0.019s sys 0.017s
I20250903 02:09:51.271193 29692 raft_consensus.cc:695] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 1 LEADER]: Becoming Leader. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:09:51.272022 29694 raft_consensus.cc:695] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 1 LEADER]: Becoming Leader. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:09:51.273264 29692 consensus_queue.cc:237] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.273350 29694 consensus_queue.cc:237] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.274469 29686 raft_consensus.cc:357] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.275240 29686 raft_consensus.cc:383] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:51.275535 29686 raft_consensus.cc:738] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cecbbd1da7046a19f694aa1fe677ed6, State: Initialized, Role: FOLLOWER
I20250903 02:09:51.278200 29686 consensus_queue.cc:260] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.281258 29687 ts_tablet_manager.cc:1428] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348: Time spent starting tablet: real 0.025s user 0.013s sys 0.001s
I20250903 02:09:51.282591 29690 tablet_bootstrap.cc:654] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:51.291533 29686 ts_tablet_manager.cc:1428] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6: Time spent starting tablet: real 0.020s user 0.004s sys 0.004s
I20250903 02:09:51.294063 29690 tablet_bootstrap.cc:492] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129: No bootstrap required, opened a new log
I20250903 02:09:51.294544 29690 ts_tablet_manager.cc:1397] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.006s
I20250903 02:09:51.297706 29690 raft_consensus.cc:357] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.298408 29690 raft_consensus.cc:383] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:51.298712 29690 raft_consensus.cc:738] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c933d23ae9746a8858c7d3ed33c3129, State: Initialized, Role: FOLLOWER
I20250903 02:09:51.299333 29690 consensus_queue.cc:260] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.301537 29690 ts_tablet_manager.cc:1428] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20250903 02:09:51.302582 29690 tablet_bootstrap.cc:492] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129: Bootstrap starting.
I20250903 02:09:51.308364 29698 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:51.309227 29690 tablet_bootstrap.cc:654] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:51.308934 29698 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.312702 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:09:51.313555 29562 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 0.
I20250903 02:09:51.314512 29698 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:09:51.315843 29604 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348; no voters:
I20250903 02:09:51.318202 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:51.319181 29484 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 0.
I20250903 02:09:51.321240 29698 raft_consensus.cc:2802] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 02:09:51.321458 29690 tablet_bootstrap.cc:492] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129: No bootstrap required, opened a new log
I20250903 02:09:51.321758 29698 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:09:51.322290 29690 ts_tablet_manager.cc:1397] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20250903 02:09:51.322368 29698 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:51.324992 29690 raft_consensus.cc:357] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.325567 29690 raft_consensus.cc:383] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:51.325799 29690 raft_consensus.cc:738] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c933d23ae9746a8858c7d3ed33c3129, State: Initialized, Role: FOLLOWER
I20250903 02:09:51.326391 29690 consensus_queue.cc:260] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.328456 29690 ts_tablet_manager.cc:1428] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250903 02:09:51.332027 29686 tablet_bootstrap.cc:492] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6: Bootstrap starting.
I20250903 02:09:51.348039 29698 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.350808 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:09:51.351583 29484 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:51.347193 29217 catalog_manager.cc:5582] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194). New cstate: current_term: 1 leader_uuid: "7b012cbd73294b4a869a1515bfd63348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: UNKNOWN } } }
I20250903 02:09:51.358285 29420 catalog_manager.cc:1261] Loaded cluster ID: 7eaaf77333364ef996cd27c71b6738f8
I20250903 02:09:51.359668 29420 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438: loading cluster ID for follower catalog manager: success
I20250903 02:09:51.359984 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
I20250903 02:09:51.360632 29562 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:51.362386 29428 catalog_manager.cc:1261] Loaded cluster ID: 7eaaf77333364ef996cd27c71b6738f8
I20250903 02:09:51.362711 29428 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: loading cluster ID for follower catalog manager: success
I20250903 02:09:51.358736 29698 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 1 election: Requested vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:09:51.359282 29686 tablet_bootstrap.cc:654] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6: Neither blocks nor log segments found. Creating new log.
I20250903 02:09:51.371080 29482 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:51.358500 29484 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 1.
I20250903 02:09:51.378630 29603 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:09:51.379513 29698 raft_consensus.cc:2802] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:09:51.380057 29698 raft_consensus.cc:695] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 LEADER]: Becoming Leader. State: Replica: 1cecbbd1da7046a19f694aa1fe677ed6, State: Running, Role: LEADER
W20250903 02:09:51.384532 29527 consensus_peers.cc:489] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 -> Peer 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481): Couldn't send request to peer 1cecbbd1da7046a19f694aa1fe677ed6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250903 02:09:51.386224 29703 consensus_queue.cc:1035] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250903 02:09:51.388336 29562 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 1.
I20250903 02:09:51.389407 29698 consensus_queue.cc:237] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.384582 29694 consensus_queue.cc:1035] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250903 02:09:51.384402 29420 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438: acquiring CA information for follower catalog manager: success
I20250903 02:09:51.406473 29428 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: acquiring CA information for follower catalog manager: success
I20250903 02:09:51.405082 29686 tablet_bootstrap.cc:492] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6: No bootstrap required, opened a new log
I20250903 02:09:51.412189 29686 ts_tablet_manager.cc:1397] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6: Time spent bootstrapping tablet: real 0.080s user 0.012s sys 0.007s
I20250903 02:09:51.408622 29217 catalog_manager.cc:5582] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194). New cstate: current_term: 1 leader_uuid: "7b012cbd73294b4a869a1515bfd63348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: UNKNOWN } } }
I20250903 02:09:51.417613 29697 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:51.418382 29697 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:51.422649 29694 consensus_queue.cc:1035] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250903 02:09:51.430753 29686 raft_consensus.cc:357] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.431881 29686 raft_consensus.cc:383] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:09:51.432303 29686 raft_consensus.cc:738] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cecbbd1da7046a19f694aa1fe677ed6, State: Initialized, Role: FOLLOWER
I20250903 02:09:51.468204 29428 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 02:09:51.437162 29686 consensus_queue.cc:260] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.470453 29686 ts_tablet_manager.cc:1428] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6: Time spent starting tablet: real 0.058s user 0.005s sys 0.002s
I20250903 02:09:51.444953 29697 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:51.471462 29698 consensus_queue.cc:1035] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250903 02:09:51.488483 29420 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 02:09:51.495667 29698 consensus_queue.cc:1035] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 02:09:51.520299 29215 catalog_manager.cc:5582] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195). New cstate: current_term: 1 leader_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: HEALTHY } } }
I20250903 02:09:51.526531 29561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:09:51.530714 29698 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:51.531388 29698 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.536403 29698 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:09:51.546837 29560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:09:51.548578 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:51.554152 29603 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:09:51.554883 29698 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250903 02:09:51.580159 29639 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:09:51.642411 29640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:51.648451 29445 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348
I20250903 02:09:51.656921 29712 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 02:09:51.898340 29562 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75"
dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
from {username='slave'} at 127.0.0.1:43738
I20250903 02:09:51.899021 29562 raft_consensus.cc:589] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 1 LEADER]: Received request to step down
I20250903 02:09:51.899381 29562 raft_consensus.cc:3053] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 02:09:51.899677 29562 raft_consensus.cc:738] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:09:51.900447 29562 consensus_queue.cc:260] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:51.901482 29562 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.002951 29712 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:09:52.003528 29712 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:52.005168 29712 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:09:52.006278 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:09:52.006278 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:52.007117 29562 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c933d23ae9746a8858c7d3ed33c3129 in term 2.
I20250903 02:09:52.008137 29444 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6
I20250903 02:09:52.008817 29712 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 02:09:52.009171 29712 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:09:52.009413 29712 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.013975 29712 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:52.015488 29712 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 election: Requested vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:09:52.016616 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
I20250903 02:09:52.016544 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:09:52.021442 29562 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c933d23ae9746a8858c7d3ed33c3129 in term 2.
I20250903 02:09:52.022465 29444 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6
I20250903 02:09:52.023149 29712 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Leader election won for term 2
I20250903 02:09:52.023630 29712 raft_consensus.cc:695] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 LEADER]: Becoming Leader. State: Replica: 9c933d23ae9746a8858c7d3ed33c3129, State: Running, Role: LEADER
I20250903 02:09:52.025024 29712 consensus_queue.cc:237] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:52.033767 29215 catalog_manager.cc:5582] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 reported cstate change: term changed from 1 to 2, leader changed from 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194) to 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193). New cstate: current_term: 2 leader_uuid: "9c933d23ae9746a8858c7d3ed33c3129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: HEALTHY } } }
I20250903 02:09:52.084794 29691 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:09:52.085403 29691 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:52.087271 29691 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:09:52.088227 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:52.088220 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:09:52.089290 29562 raft_consensus.cc:2391] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cecbbd1da7046a19f694aa1fe677ed6 in current term 2: Already voted for candidate 9c933d23ae9746a8858c7d3ed33c3129 in this term.
I20250903 02:09:52.090126 29604 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:09:52.090763 29691 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.095458 29691 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 02:09:52.106590 29562 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9"
dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
from {username='slave'} at 127.0.0.1:43738
I20250903 02:09:52.107249 29562 raft_consensus.cc:589] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 1 LEADER]: Received request to step down
I20250903 02:09:52.107633 29562 raft_consensus.cc:3053] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 02:09:52.107954 29562 raft_consensus.cc:738] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:09:52.108806 29562 consensus_queue.cc:260] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.110167 29562 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.116433 29639 raft_consensus.cc:1273] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Refusing update from remote peer 9c933d23ae9746a8858c7d3ed33c3129: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250903 02:09:52.117635 29712 consensus_queue.cc:1035] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250903 02:09:52.128607 29562 raft_consensus.cc:1273] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Refusing update from remote peer 9c933d23ae9746a8858c7d3ed33c3129: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250903 02:09:52.131253 29697 consensus_queue.cc:1035] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250903 02:09:52.228569 29691 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:09:52.229836 29691 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.232434 29739 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:09:52.232964 29739 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.234521 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:09:52.235625 29739 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:52.236162 29561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:09:52.237779 29691 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:09:52.235572 29562 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 2.
I20250903 02:09:52.239351 29604 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348; no voters:
I20250903 02:09:52.241037 29695 raft_consensus.cc:2802] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 02:09:52.241457 29695 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:09:52.241822 29695 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.254488 29695 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.257558 29561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
I20250903 02:09:52.260697 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:52.261631 29484 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 1.
I20250903 02:09:52.266103 29561 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 2.
I20250903 02:09:52.267887 29604 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348; no voters:
I20250903 02:09:52.269287 29691 raft_consensus.cc:2802] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Leader election won for term 2
I20250903 02:09:52.364550 29702 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cecbbd1da7046a19f694aa1fe677ed6)
I20250903 02:09:52.365280 29702 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.366262 29740 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1cecbbd1da7046a19f694aa1fe677ed6)
I20250903 02:09:52.366968 29740 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.371693 29561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:09:52.372598 29561 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c933d23ae9746a8858c7d3ed33c3129 in term 1.
I20250903 02:09:52.373585 29740 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:52.373672 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:52.382355 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:52.383283 29484 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 1.
I20250903 02:09:52.384709 29525 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6
I20250903 02:09:52.374392 29702 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:52.387398 29702 raft_consensus.cc:2802] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 02:09:52.387897 29702 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cecbbd1da7046a19f694aa1fe677ed6)
I20250903 02:09:52.388367 29702 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.375006 29444 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:09:52.398367 29740 raft_consensus.cc:2802] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 02:09:52.398959 29740 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1cecbbd1da7046a19f694aa1fe677ed6)
I20250903 02:09:52.399520 29740 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.256157 29695 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 2 election: Requested vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:09:52.432597 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:09:52.439671 29481 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.443970 29740 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.432479 29695 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c933d23ae9746a8858c7d3ed33c3129)
I20250903 02:09:52.445945 29695 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:52.447258 29481 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 2.
I20250903 02:09:52.448086 29695 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:09:52.457020 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:52.458459 29740 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 election: Requested vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
W20250903 02:09:52.455559 29525 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.35.193:41955, user_credentials={real_user=slave}} blocked reactor thread for 69119.2us
I20250903 02:09:52.464475 29561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
I20250903 02:09:52.460263 29484 raft_consensus.cc:1238] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Rejecting Update request from peer 1cecbbd1da7046a19f694aa1fe677ed6 for earlier term 1. Current term is 2. Ops: []
I20250903 02:09:52.466634 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:09:52.465574 29742 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e1ec9fa14084d698da867035b48fc13)
I20250903 02:09:52.470813 29742 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:52.471354 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:09:52.471258 29702 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.472481 29560 raft_consensus.cc:1238] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Rejecting Update request from peer 1cecbbd1da7046a19f694aa1fe677ed6 for earlier term 1. Current term is 2. Ops: []
I20250903 02:09:52.459692 29691 raft_consensus.cc:695] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 LEADER]: Becoming Leader. State: Replica: 1cecbbd1da7046a19f694aa1fe677ed6, State: Running, Role: LEADER
I20250903 02:09:52.473699 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:52.475085 29742 leader_election.cc:290] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649)
I20250903 02:09:52.474086 29691 consensus_queue.cc:237] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.482621 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:09:52.482705 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:09:52.483842 29484 raft_consensus.cc:2391] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b012cbd73294b4a869a1515bfd63348 in current term 2: Already voted for candidate 9c933d23ae9746a8858c7d3ed33c3129 in this term.
I20250903 02:09:52.485103 29525 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348; no voters: 1cecbbd1da7046a19f694aa1fe677ed6, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:09:52.485854 29743 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250903 02:09:52.483333 29702 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 2 election: Requested vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:52.496253 29444 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348
I20250903 02:09:52.498170 29640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:52.503311 29691 consensus_queue.cc:1046] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250903 02:09:52.504973 29741 raft_consensus.cc:3053] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 02:09:52.505517 29741 raft_consensus.cc:738] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1cecbbd1da7046a19f694aa1fe677ed6, State: Running, Role: LEADER
I20250903 02:09:52.506436 29741 consensus_queue.cc:260] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.507608 29741 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 1 FOLLOWER]: Advancing to term 2
W20250903 02:09:52.525799 29606 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250903 02:09:52.541261 29697 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250903 02:09:52.551501 29603 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.35.193:41955, user_credentials={real_user=slave}} blocked reactor thread for 73532.3us
I20250903 02:09:52.553929 29702 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:52.554535 29702 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.556092 29603 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:09:52.557566 29702 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:52.558310 29640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:52.560179 29741 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250903 02:09:52.565819 29747 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e1ec9fa14084d698da867035b48fc13)
I20250903 02:09:52.567438 29747 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:52.570405 29747 leader_election.cc:290] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587)
I20250903 02:09:52.654320 29227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "6e1ec9fa14084d698da867035b48fc13" is_pre_election: true
I20250903 02:09:52.680625 29227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36f57bf97db44ab1a86d32e59b4ad438" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "6e1ec9fa14084d698da867035b48fc13" is_pre_election: true
I20250903 02:09:52.680864 29291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "36f57bf97db44ab1a86d32e59b4ad438" is_pre_election: true
I20250903 02:09:52.681557 29291 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f21cd83f94b4a75ac8648f471aeb85b in term 1.
I20250903 02:09:52.682380 29357 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36f57bf97db44ab1a86d32e59b4ad438" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" is_pre_election: true
I20250903 02:09:52.682971 29330 leader_election.cc:304] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438, 6f21cd83f94b4a75ac8648f471aeb85b; no voters: 6e1ec9fa14084d698da867035b48fc13
I20250903 02:09:52.683820 29742 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 02:09:52.684165 29742 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e1ec9fa14084d698da867035b48fc13)
I20250903 02:09:52.684531 29742 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.684484 29266 leader_election.cc:304] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438; no voters: 6e1ec9fa14084d698da867035b48fc13, 6f21cd83f94b4a75ac8648f471aeb85b
I20250903 02:09:52.685431 29747 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 02:09:52.691481 29742 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:52.693511 29742 leader_election.cc:290] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 2 election: Requested vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649)
I20250903 02:09:52.694768 29227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "6e1ec9fa14084d698da867035b48fc13"
I20250903 02:09:52.696465 29291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "36f57bf97db44ab1a86d32e59b4ad438"
I20250903 02:09:52.697098 29291 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.705054 29291 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f21cd83f94b4a75ac8648f471aeb85b in term 2.
I20250903 02:09:52.705809 29292 raft_consensus.cc:1238] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 2 FOLLOWER]: Rejecting Update request from peer 6e1ec9fa14084d698da867035b48fc13 for earlier term 1. Current term is 2. Ops: []
I20250903 02:09:52.706477 29330 leader_election.cc:304] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438, 6f21cd83f94b4a75ac8648f471aeb85b; no voters: 6e1ec9fa14084d698da867035b48fc13
I20250903 02:09:52.707405 29742 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 2 FOLLOWER]: Leader election won for term 2
I20250903 02:09:52.708626 29741 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:52.709309 29741 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.714674 29741 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:09:52.715886 29742 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 2 LEADER]: Becoming Leader. State: Replica: 6f21cd83f94b4a75ac8648f471aeb85b, State: Running, Role: LEADER
I20250903 02:09:52.716899 29742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:52.718895 29640 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9"
dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
from {username='slave'} at 127.0.0.1:40666
I20250903 02:09:52.719483 29640 raft_consensus.cc:589] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 LEADER]: Received request to step down
I20250903 02:09:52.719830 29640 raft_consensus.cc:3053] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 LEADER]: Stepping down as leader of term 2
I20250903 02:09:52.720131 29640 raft_consensus.cc:738] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1cecbbd1da7046a19f694aa1fe677ed6, State: Running, Role: LEADER
I20250903 02:09:52.720840 29640 consensus_queue.cc:260] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.721778 29640 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:52.723261 29560 raft_consensus.cc:1273] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Refusing update from remote peer 1cecbbd1da7046a19f694aa1fe677ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250903 02:09:52.725550 29743 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9c933d23ae9746a8858c7d3ed33c3129)
I20250903 02:09:52.726033 29743 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:52.727926 29743 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:09:52.728412 29743 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:09:52.728832 29743 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.729126 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:52.729256 29640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:52.730129 29640 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 2.
I20250903 02:09:52.731217 29527 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348; no voters: 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:09:52.732177 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:52.732865 29638 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 2.
I20250903 02:09:52.734005 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:52.734834 29481 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 2.
I20250903 02:09:52.737298 29525 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:09:52.739342 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:09:52.740021 29482 raft_consensus.cc:2374] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 2. Re-sending same reply.
I20250903 02:09:52.733006 29743 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:52.742314 29743 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250903 02:09:52.744290 29752 raft_consensus.cc:2802] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250903 02:09:52.744683 29752 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:09:52.745026 29752 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:52.743317 29743 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9c933d23ae9746a8858c7d3ed33c3129)
I20250903 02:09:52.746548 29743 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:52.750221 29738 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 }, Status: INVALID_TERM, Last received: 1.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250903 02:09:52.750738 29752 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.751345 29447 raft_consensus.cc:3053] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 02:09:52.751719 29447 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e1ec9fa14084d698da867035b48fc13, State: Running, Role: LEADER
I20250903 02:09:52.752698 29447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:52.754032 29447 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:09:52.754209 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
W20250903 02:09:52.751698 29639 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40650 (request call id 7) took 295 ms (client timeout 283 ms). Trace:
I20250903 02:09:52.754885 29482 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Advancing to term 3
W20250903 02:09:52.754899 29639 rpcz_store.cc:269] 0903 02:09:52.456595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 02:09:52.456727 (+ 132us) service_pool.cc:225] Handling call
0903 02:09:52.751494 (+294767us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b5400330254. stack: 00007fef2702c473 00007fef2702c370 00007fef26982574 00000000003bff74 00000000003bff09 00007fef2d88b6e1 00007fef2d90b2aa 00007fef2f8c85c7 00007fef2d45b0c4 00007fef2d45b052 00007fef2d45af81 00007fef2d45aefc 00007fef2d45a1b2 00007fef2bba0ddc 00007fef2bba0216 00007fef2bb9fbbf
0903 02:09:52.751644 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113246976}
I20250903 02:09:52.759495 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:09:52.760267 29638 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:52.762229 29482 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 3.
I20250903 02:09:52.763108 29752 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 election: Requested vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:52.764632 29525 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:09:52.765503 29752 raft_consensus.cc:2802] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 3 FOLLOWER]: Leader election won for term 3
I20250903 02:09:52.766065 29752 raft_consensus.cc:695] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 3 LEADER]: Becoming Leader. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:09:52.766981 29752 consensus_queue.cc:237] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.767445 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:09:52.772454 29743 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:52.773715 29560 raft_consensus.cc:1238] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 3 FOLLOWER]: Rejecting Update request from peer 9c933d23ae9746a8858c7d3ed33c3129 for earlier term 2. Current term is 3. Ops: []
I20250903 02:09:52.774849 29638 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 3.
I20250903 02:09:52.778515 29740 consensus_queue.cc:1046] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250903 02:09:52.779733 29740 raft_consensus.cc:3053] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 LEADER]: Stepping down as leader of term 2
W20250903 02:09:52.773762 29603 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Timed out: RequestConsensusVote RPC to 127.27.35.193:41955 timed out after 0.200s (SENT)
I20250903 02:09:52.780570 29740 raft_consensus.cc:738] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9c933d23ae9746a8858c7d3ed33c3129, State: Running, Role: LEADER
I20250903 02:09:52.780613 29751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f21cd83f94b4a75ac8648f471aeb85b. Latest consensus state: current_term: 2 leader_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:52.781491 29751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:52.782409 29740 consensus_queue.cc:260] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:52.784299 29740 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:52.785194 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:09:52.785917 29743 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 election: Requested vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:09:52.787880 29637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:09:52.788985 29484 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 2.
I20250903 02:09:52.786957 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:09:52.789294 29637 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:52.785418 29640 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:52.790436 29525 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6
I20250903 02:09:52.790791 29640 raft_consensus.cc:2366] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c933d23ae9746a8858c7d3ed33c3129 for earlier term 2. Current term is 3.
I20250903 02:09:52.791602 29743 raft_consensus.cc:2802] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250903 02:09:52.792029 29743 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:09:52.792366 29743 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:52.792600 29445 leader_election.cc:400] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481) with higher term. Message: Invalid argument: T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c933d23ae9746a8858c7d3ed33c3129 for earlier term 2. Current term is 3.
I20250903 02:09:52.794564 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:09:52.795327 29639 raft_consensus.cc:2366] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c933d23ae9746a8858c7d3ed33c3129 for earlier term 2. Current term is 3.
I20250903 02:09:52.797340 29445 leader_election.cc:400] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481) with higher term. Message: Invalid argument: T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c933d23ae9746a8858c7d3ed33c3129 for earlier term 2. Current term is 3.
I20250903 02:09:52.797668 29445 leader_election.cc:403] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250903 02:09:52.784291 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
W20250903 02:09:52.798731 29443 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250903 02:09:52.798791 29697 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:52.799099 29482 raft_consensus.cc:2391] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cecbbd1da7046a19f694aa1fe677ed6 in current term 3: Already voted for candidate 7b012cbd73294b4a869a1515bfd63348 in this term.
I20250903 02:09:52.800813 29603 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:09:52.801694 29741 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250903 02:09:52.801587 29637 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 3.
I20250903 02:09:52.803076 29638 raft_consensus.cc:1238] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Rejecting Update request from peer 9c933d23ae9746a8858c7d3ed33c3129 for earlier term 2. Current term is 3. Ops: []
I20250903 02:09:52.803239 29527 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348; no voters:
I20250903 02:09:52.805447 29752 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 3 FOLLOWER]: Leader election won for term 3
I20250903 02:09:52.806674 29752 raft_consensus.cc:695] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 3 LEADER]: Becoming Leader. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:09:52.807624 29752 consensus_queue.cc:237] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:09:52.808151 29743 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.810899 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:09:52.811245 29697 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481) with higher term. Message: Invalid argument: T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c933d23ae9746a8858c7d3ed33c3129 for earlier term 2. Current term is 3.
I20250903 02:09:52.811901 29743 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 election: Requested vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:52.812134 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:09:52.813043 29638 raft_consensus.cc:2408] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7b012cbd73294b4a869a1515bfd63348 for term 3 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20250903 02:09:52.814010 29753 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:09:52.818373 29227 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 2 FOLLOWER]: Refusing update from remote peer 6f21cd83f94b4a75ac8648f471aeb85b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250903 02:09:52.818933 29292 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 2 FOLLOWER]: Refusing update from remote peer 6f21cd83f94b4a75ac8648f471aeb85b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250903 02:09:52.819880 29754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250903 02:09:52.820688 29742 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250903 02:09:52.830399 29747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f21cd83f94b4a75ac8648f471aeb85b. Latest consensus state: current_term: 2 leader_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:52.830883 29482 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 3.
I20250903 02:09:52.831162 29747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:52.832108 29447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f21cd83f94b4a75ac8648f471aeb85b. Latest consensus state: current_term: 2 leader_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:52.832768 29447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:52.832998 29525 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6
I20250903 02:09:52.834192 29752 raft_consensus.cc:2802] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 3 FOLLOWER]: Leader election won for term 3
I20250903 02:09:52.836237 29481 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 3.
I20250903 02:09:52.838593 29742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:52.839707 29742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:52.839421 29755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:52.835352 29752 raft_consensus.cc:695] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 3 LEADER]: Becoming Leader. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:09:52.840970 29755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:52.841727 29752 consensus_queue.cc:237] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:09:52.846132 29447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:52.847234 29447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:52.849915 29747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:52.850732 29747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:52.852698 29753 catalog_manager.cc:671] Loaded metadata for table test-table [id=9d7b841db0334b5897236fd155f908b0]
I20250903 02:09:52.857367 29481 raft_consensus.cc:1273] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Refusing update from remote peer 7b012cbd73294b4a869a1515bfd63348: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250903 02:09:52.859423 29752 consensus_queue.cc:1035] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250903 02:09:52.859869 29753 tablet_loader.cc:96] loaded metadata for tablet 2da3487154804c71a24242048c1b0c16 (table test-table [id=9d7b841db0334b5897236fd155f908b0])
I20250903 02:09:52.861233 29753 tablet_loader.cc:96] loaded metadata for tablet beaf5654ab144e23a627269917ad5ee9 (table test-table [id=9d7b841db0334b5897236fd155f908b0])
I20250903 02:09:52.862475 29753 tablet_loader.cc:96] loaded metadata for tablet db760726eb304ef0b69159db4bbaef75 (table test-table [id=9d7b841db0334b5897236fd155f908b0])
I20250903 02:09:52.862787 29638 raft_consensus.cc:1273] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Refusing update from remote peer 7b012cbd73294b4a869a1515bfd63348: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250903 02:09:52.863938 29753 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:09:52.864323 29752 consensus_queue.cc:1035] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250903 02:09:52.868148 29753 catalog_manager.cc:1261] Loaded cluster ID: 7eaaf77333364ef996cd27c71b6738f8
I20250903 02:09:52.868479 29753 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:09:52.872800 29753 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:09:52.877295 29753 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: Loaded TSK: 0
I20250903 02:09:52.878881 29753 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:09:52.892043 29347 catalog_manager.cc:5582] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 reported cstate change: term changed from 1 to 3, leader changed from 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195) to 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194). New cstate: current_term: 3 leader_uuid: "7b012cbd73294b4a869a1515bfd63348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } }
I20250903 02:09:52.923705 29481 raft_consensus.cc:1273] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Refusing update from remote peer 7b012cbd73294b4a869a1515bfd63348: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250903 02:09:52.924805 29638 raft_consensus.cc:1273] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Refusing update from remote peer 7b012cbd73294b4a869a1515bfd63348: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250903 02:09:52.927985 29743 consensus_queue.cc:1035] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250903 02:09:52.930284 29665 heartbeater.cc:499] Master 127.27.35.252:45587 was elected leader, sending a full tablet report...
I20250903 02:09:52.931102 29509 heartbeater.cc:499] Master 127.27.35.252:45587 was elected leader, sending a full tablet report...
I20250903 02:09:52.932122 29587 heartbeater.cc:499] Master 127.27.35.252:45587 was elected leader, sending a full tablet report...
I20250903 02:09:52.931994 29743 consensus_queue.cc:1035] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250903 02:09:52.941434 29637 raft_consensus.cc:1273] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Refusing update from remote peer 7b012cbd73294b4a869a1515bfd63348: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20250903 02:09:52.941879 29637 pending_rounds.cc:85] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6: Aborting all ops after (but not including) 2
I20250903 02:09:52.942209 29637 pending_rounds.cc:107] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6: Aborting uncommitted NO_OP operation due to leader change: 2.3
I20250903 02:09:52.942509 29637 raft_consensus.cc:2887] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250903 02:09:52.944345 29752 consensus_queue.cc:1035] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250903 02:09:52.946187 29346 catalog_manager.cc:5582] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "7b012cbd73294b4a869a1515bfd63348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: UNKNOWN } } }
I20250903 02:09:52.956185 29346 catalog_manager.cc:5582] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 reported cstate change: term changed from 2 to 3, leader changed from 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193) to 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194). New cstate: current_term: 3 leader_uuid: "7b012cbd73294b4a869a1515bfd63348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: UNKNOWN } } }
I20250903 02:09:52.964504 29481 raft_consensus.cc:1273] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Refusing update from remote peer 7b012cbd73294b4a869a1515bfd63348: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250903 02:09:52.971529 29702 consensus_queue.cc:1035] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250903 02:09:53.010249 29400 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 02:09:53.032038 29345 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43792:
table { table_id: "9d7b841db0334b5897236fd155f908b0" table_name: "test-table" } new_table_name: "NewTableName"
I20250903 02:09:53.057296 27791 master.cc:561] Master@127.27.35.252:45587 shutting down...
I20250903 02:09:53.106952 29783 tablet.cc:1722] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 02:09:53.132551 29780 tablet.cc:1722] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 02:09:53.137920 29785 tablet.cc:1722] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 02:09:53.138051 29786 tablet.cc:1722] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 02:09:53.141650 29787 tablet.cc:1722] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 02:09:53.134955 29779 tablet.cc:1722] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 02:09:53.158218 29784 tablet.cc:1722] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 02:09:53.155612 29782 tablet.cc:1722] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 02:09:53.175946 29783 tablet.cc:1722] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250903 02:09:53.228150 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 2 LEADER]: Raft consensus shutting down.
I20250903 02:09:53.229215 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 02:09:53.229782 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: stopping tablet replica
I20250903 02:09:53.245018 27791 master.cc:583] Master@127.27.35.252:45587 shutdown complete.
I20250903 02:09:53.360041 29791 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6f21cd83f94b4a75ac8648f471aeb85b)
I20250903 02:09:53.360669 29792 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6f21cd83f94b4a75ac8648f471aeb85b)
I20250903 02:09:53.360539 29791 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:53.361117 29792 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:53.362183 29791 leader_election.cc:290] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649), 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587)
I20250903 02:09:53.362870 29792 leader_election.cc:290] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587)
I20250903 02:09:53.363152 29292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e1ec9fa14084d698da867035b48fc13" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "36f57bf97db44ab1a86d32e59b4ad438" is_pre_election: true
I20250903 02:09:53.363842 29292 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1ec9fa14084d698da867035b48fc13 in term 2.
I20250903 02:09:53.365206 29201 leader_election.cc:304] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438, 6e1ec9fa14084d698da867035b48fc13; no voters:
I20250903 02:09:53.366041 29791 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250903 02:09:53.366473 29791 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6f21cd83f94b4a75ac8648f471aeb85b)
I20250903 02:09:53.366873 29791 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:53.366626 29227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36f57bf97db44ab1a86d32e59b4ad438" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "6e1ec9fa14084d698da867035b48fc13" is_pre_election: true
W20250903 02:09:53.368891 29266 leader_election.cc:336] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587): Network error: Client connection negotiation failed: client connection to 127.27.35.252:45587: connect: Connection refused (error 111)
W20250903 02:09:53.370609 29201 leader_election.cc:336] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587): Network error: Client connection negotiation failed: client connection to 127.27.35.252:45587: connect: Connection refused (error 111)
I20250903 02:09:53.372615 29791 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:53.373528 29227 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36f57bf97db44ab1a86d32e59b4ad438 in current term 3: Already voted for candidate 6e1ec9fa14084d698da867035b48fc13 in this term.
I20250903 02:09:53.374094 29791 leader_election.cc:290] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [CANDIDATE]: Term 3 election: Requested vote from peers 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649), 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587)
I20250903 02:09:53.374832 29292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6e1ec9fa14084d698da867035b48fc13" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "36f57bf97db44ab1a86d32e59b4ad438"
I20250903 02:09:53.375377 29292 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:53.377072 29266 leader_election.cc:304] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438; no voters: 6e1ec9fa14084d698da867035b48fc13, 6f21cd83f94b4a75ac8648f471aeb85b
W20250903 02:09:53.378500 29201 leader_election.cc:336] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587): Network error: Client connection negotiation failed: client connection to 127.27.35.252:45587: connect: Connection refused (error 111)
I20250903 02:09:53.380990 29292 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e1ec9fa14084d698da867035b48fc13 in term 3.
I20250903 02:09:53.381402 29792 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250903 02:09:53.381861 29201 leader_election.cc:304] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438, 6e1ec9fa14084d698da867035b48fc13; no voters: 6f21cd83f94b4a75ac8648f471aeb85b
I20250903 02:09:53.382668 29791 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 3 FOLLOWER]: Leader election won for term 3
I20250903 02:09:53.383914 29791 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 3 LEADER]: Becoming Leader. State: Replica: 6e1ec9fa14084d698da867035b48fc13, State: Running, Role: LEADER
I20250903 02:09:53.384550 29791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:53.387676 29797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e1ec9fa14084d698da867035b48fc13. Latest consensus state: current_term: 3 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:53.388266 29797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:53.389559 29798 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:09:53.395581 29798 catalog_manager.cc:671] Loaded metadata for table NewTableName [id=9d7b841db0334b5897236fd155f908b0]
I20250903 02:09:53.400734 29798 tablet_loader.cc:96] loaded metadata for tablet 2da3487154804c71a24242048c1b0c16 (table NewTableName [id=9d7b841db0334b5897236fd155f908b0])
I20250903 02:09:53.401679 29798 tablet_loader.cc:96] loaded metadata for tablet beaf5654ab144e23a627269917ad5ee9 (table NewTableName [id=9d7b841db0334b5897236fd155f908b0])
I20250903 02:09:53.402522 29798 tablet_loader.cc:96] loaded metadata for tablet db760726eb304ef0b69159db4bbaef75 (table NewTableName [id=9d7b841db0334b5897236fd155f908b0])
I20250903 02:09:53.403527 29798 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:09:53.406644 29798 catalog_manager.cc:1261] Loaded cluster ID: 7eaaf77333364ef996cd27c71b6738f8
I20250903 02:09:53.406867 29798 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:09:53.410334 29798 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:09:53.413467 29798 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13: Loaded TSK: 0
I20250903 02:09:53.414319 29798 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:09:53.459005 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:09:53.465078 29799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:09:53.465134 29800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:53.467996 27791 server_base.cc:1047] running on GCE node
W20250903 02:09:53.468048 29802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:09:53.469192 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:09:53.469417 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:09:53.469617 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865393469591 us; error 0 us; skew 500 ppm
I20250903 02:09:53.470163 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:09:53.472630 27791 webserver.cc:480] Webserver started at http://127.27.35.252:41231/ using document root <none> and password file <none>
I20250903 02:09:53.473157 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:09:53.473387 27791 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250903 02:09:53.477286 27791 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20250903 02:09:53.480358 29807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:09:53.481189 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 02:09:53.481462 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-2-root
uuid: "6f21cd83f94b4a75ac8648f471aeb85b"
format_stamp: "Formatted at 2025-09-03 02:09:50 on dist-test-slave-h78g"
I20250903 02:09:53.481721 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250903 02:09:53.499634 29201 consensus_peers.cc:489] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 -> Peer 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587): Couldn't send request to peer 6f21cd83f94b4a75ac8648f471aeb85b. Status: Network error: Client connection negotiation failed: client connection to 127.27.35.252:45587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 02:09:53.505261 29292 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Refusing update from remote peer 6e1ec9fa14084d698da867035b48fc13: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250903 02:09:53.506228 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:09:53.506629 29797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250903 02:09:53.507314 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:09:53.509869 29792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e1ec9fa14084d698da867035b48fc13. Latest consensus state: current_term: 3 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:53.510646 29792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:53.514389 29797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:53.515130 29797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:53.515025 29792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:53.515815 29792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:53.527990 29667 heartbeater.cc:499] Master 127.27.35.254:41877 was elected leader, sending a full tablet report...
I20250903 02:09:53.536638 29511 heartbeater.cc:499] Master 127.27.35.254:41877 was elected leader, sending a full tablet report...
I20250903 02:09:53.545666 29589 heartbeater.cc:499] Master 127.27.35.254:41877 was elected leader, sending a full tablet report...
I20250903 02:09:53.556835 29217 catalog_manager.cc:6127] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13: NewTableName [id=9d7b841db0334b5897236fd155f908b0] alter complete (version 1)
I20250903 02:09:53.581756 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.252:45587
I20250903 02:09:53.581857 29861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.252:45587 every 8 connection(s)
I20250903 02:09:53.590787 29862 sys_catalog.cc:263] Verifying existing consensus state
W20250903 02:09:53.592705 29313 debug-util.cc:398] Leaking SignalData structure 0x7b08000a6b00 after lost signal to thread 27794
W20250903 02:09:53.593622 29313 debug-util.cc:398] Leaking SignalData structure 0x7b080009c420 after lost signal to thread 29251
W20250903 02:09:53.594353 29313 debug-util.cc:398] Leaking SignalData structure 0x7b080002ba80 after lost signal to thread 29316
I20250903 02:09:53.595780 29862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: Bootstrap starting.
W20250903 02:09:53.596168 29313 debug-util.cc:398] Leaking SignalData structure 0x7b080021fae0 after lost signal to thread 29508
W20250903 02:09:53.597585 29313 debug-util.cc:398] Leaking SignalData structure 0x7b0800235320 after lost signal to thread 29664
I20250903 02:09:53.630595 29665 heartbeater.cc:344] Connected to a master server at 127.27.35.252:45587
I20250903 02:09:53.646297 29509 heartbeater.cc:344] Connected to a master server at 127.27.35.252:45587
I20250903 02:09:53.662235 29862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 02:09:53.663582 29862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: Bootstrap complete.
I20250903 02:09:53.667059 29862 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:53.667886 29862 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f21cd83f94b4a75ac8648f471aeb85b, State: Initialized, Role: FOLLOWER
I20250903 02:09:53.668742 29862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:09:53.671239 29587 heartbeater.cc:344] Connected to a master server at 127.27.35.252:45587
I20250903 02:09:53.677618 29862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:09:53.678017 29837 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 2 FOLLOWER]: Advancing to term 3
I20250903 02:09:53.676574 29866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:53.687840 29866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:53.698546 29871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e1ec9fa14084d698da867035b48fc13. Latest consensus state: current_term: 3 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:53.699407 29871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:53.700779 29866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:53.701812 29866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [sys.catalog]: This master's current role is: FOLLOWER
I20250903 02:09:53.711863 29881 catalog_manager.cc:1261] Loaded cluster ID: 7eaaf77333364ef996cd27c71b6738f8
I20250903 02:09:53.712226 29881 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: loading cluster ID for follower catalog manager: success
I20250903 02:09:53.718025 29881 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: acquiring CA information for follower catalog manager: success
I20250903 02:09:53.718559 29814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6e1ec9fa14084d698da867035b48fc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } } }
I20250903 02:09:53.719419 29814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [sys.catalog]: This master's current role is: LEADER
I20250903 02:09:53.723151 29881 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250903 02:09:53.772217 29824 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" instance_seqno: 1756865390475507) as {username='slave'} at 127.0.0.1:43978; Asking this server to re-register.
I20250903 02:09:53.773059 29825 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" instance_seqno: 1756865390822149) as {username='slave'} at 127.0.0.1:43976; Asking this server to re-register.
I20250903 02:09:53.774121 29509 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:53.775060 29665 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:53.777856 29824 ts_manager.cc:194] Registered new tserver with Master: 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:09:53.779342 29825 ts_manager.cc:194] Registered new tserver with Master: 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:09:53.782147 29825 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" instance_seqno: 1756865390635330) as {username='slave'} at 127.0.0.1:43990; Asking this server to re-register.
I20250903 02:09:53.783778 29587 heartbeater.cc:461] Registering TS with master...
I20250903 02:09:53.787544 29825 ts_manager.cc:194] Registered new tserver with Master: 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:09:53.788305 27791 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.203452908s
I20250903 02:10:20.198689 29908 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:20.270678 29908 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:20.484227 29909 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:20.485111 29909 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:20.487915 29909 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:10:20.583818 29907 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:21.194201 29908 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:10:21.206358 29910 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e1ec9fa14084d698da867035b48fc13)
I20250903 02:10:21.207633 29907 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:21.208124 29910 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:10:21.210031 29907 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:10:21.225451 29910 leader_election.cc:290] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649)
I20250903 02:10:21.630924 29915 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e1ec9fa14084d698da867035b48fc13)
I20250903 02:10:21.631614 29915 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:10:21.635274 29915 leader_election.cc:290] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587)
W20250903 02:10:21.638177 29606 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.698s user 0.011s sys 0.016s
W20250903 02:10:21.638792 29606 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.698s user 0.012s sys 0.016s
I20250903 02:10:21.644536 29292 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35358 (request call id 304) took 1838 ms. Trace:
I20250903 02:10:21.645221 29292 rpcz_store.cc:276] 0903 02:10:19.806054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0903 02:10:21.643513 (+1837459us) service_pool.cc:225] Handling call
0903 02:10:21.643834 (+ 321us) raft_consensus.cc:1469] Updating replica for 0 ops
0903 02:10:21.644047 (+ 213us) raft_consensus.cc:1532] Early marking committed up to index 0
0903 02:10:21.644077 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0903 02:10:21.644123 (+ 46us) raft_consensus.cc:1643] Marking committed up to 16
0903 02:10:21.644158 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0903 02:10:21.644214 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0903 02:10:21.644356 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250903 02:10:21.649464 29604 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.731s user 0.023s sys 0.024s
W20250903 02:10:21.649976 29604 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.744s user 0.023s sys 0.024s
I20250903 02:10:21.655031 29560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:21.666953 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:21.673599 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:21.676357 29227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36f57bf97db44ab1a86d32e59b4ad438" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "6e1ec9fa14084d698da867035b48fc13" is_pre_election: true
I20250903 02:10:21.676985 29912 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:21.677821 29912 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:21.680262 29913 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:21.681725 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:10:21.681947 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:10:21.682461 29484 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 3.
I20250903 02:10:21.682096 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:10:21.683348 29482 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 3.
I20250903 02:10:21.681085 29913 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:21.693663 29913 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:10:21.694376 29913 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:21.695046 29913 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:21.696470 29560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:21.696237 29912 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:21.702406 29808 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.943s user 0.000s sys 0.026s
W20250903 02:10:21.702894 29808 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.944s user 0.000s sys 0.027s
I20250903 02:10:21.704197 29481 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 3.
I20250903 02:10:21.707121 29603 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 9c933d23ae9746a8858c7d3ed33c3129; no voters: 7b012cbd73294b4a869a1515bfd63348
I20250903 02:10:21.708666 29907 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250903 02:10:21.707228 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:10:21.709162 29907 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:21.709728 29907 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:10:21.710469 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:10:21.708819 29603 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:10:21.716821 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:21.727890 29923 raft_consensus.cc:2802] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250903 02:10:21.729162 29923 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:21.730038 29923 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:10:21.735255 29445 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348
I20250903 02:10:21.737625 29603 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 9c933d23ae9746a8858c7d3ed33c3129; no voters: 7b012cbd73294b4a869a1515bfd63348
I20250903 02:10:21.740171 29924 raft_consensus.cc:2802] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250903 02:10:21.742662 29912 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 02:10:21.748169 29924 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:21.749168 29924 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:10:21.752452 29913 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:10:21.753505 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:21.758580 29637 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:10:21.767354 29923 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:21.773195 29637 raft_consensus.cc:2391] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c933d23ae9746a8858c7d3ed33c3129 in current term 4: Already voted for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in this term.
I20250903 02:10:21.774220 29923 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 election: Requested vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:10:21.781037 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
I20250903 02:10:21.781214 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:10:21.781842 29481 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:10:21.782202 29445 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348
I20250903 02:10:21.786371 29637 raft_consensus.cc:1238] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Rejecting Update request from peer 7b012cbd73294b4a869a1515bfd63348 for earlier term 3. Current term is 4. Ops: []
I20250903 02:10:21.788093 29914 consensus_queue.cc:1046] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250903 02:10:21.789323 29914 raft_consensus.cc:3053] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 3 LEADER]: Stepping down as leader of term 3
I20250903 02:10:21.789726 29914 raft_consensus.cc:738] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:10:21.790539 29914 consensus_queue.cc:260] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:21.791846 29914 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:10:21.794664 29481 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 4.
I20250903 02:10:21.801317 29913 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 02:10:21.794507 29907 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
W20250903 02:10:21.798226 29528 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250903 02:10:21.807689 29603 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 9c933d23ae9746a8858c7d3ed33c3129; no voters: 7b012cbd73294b4a869a1515bfd63348
I20250903 02:10:21.807737 29482 raft_consensus.cc:1238] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Rejecting Update request from peer 7b012cbd73294b4a869a1515bfd63348 for earlier term 3. Current term is 4. Ops: []
I20250903 02:10:21.819691 29923 raft_consensus.cc:2802] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Leader election won for term 4
I20250903 02:10:21.821902 29638 raft_consensus.cc:2391] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c933d23ae9746a8858c7d3ed33c3129 in current term 4: Already voted for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in this term.
I20250903 02:10:21.823385 29639 raft_consensus.cc:1238] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Rejecting Update request from peer 7b012cbd73294b4a869a1515bfd63348 for earlier term 3. Current term is 4. Ops: []
I20250903 02:10:21.824625 29445 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348
I20250903 02:10:21.826232 29913 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:10:21.828867 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
I20250903 02:10:21.829293 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:10:21.831185 29603 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:21.833303 29923 raft_consensus.cc:695] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 LEADER]: Becoming Leader. State: Replica: 1cecbbd1da7046a19f694aa1fe677ed6, State: Running, Role: LEADER
I20250903 02:10:21.834553 29907 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 election: Requested vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:10:21.835551 29907 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250903 02:10:21.836673 29914 consensus_queue.cc:1046] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20250903 02:10:21.852839 29914 raft_consensus.cc:3053] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 3 LEADER]: Stepping down as leader of term 3
I20250903 02:10:21.853474 29914 raft_consensus.cc:738] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:10:21.854714 29914 consensus_queue.cc:260] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
W20250903 02:10:21.865278 29525 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.35.193:41955, user_credentials={real_user=slave}} blocked reactor thread for 61547.2us
I20250903 02:10:21.867646 29837 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36f57bf97db44ab1a86d32e59b4ad438" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" is_pre_election: true
I20250903 02:10:21.856396 29923 consensus_queue.cc:237] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:21.870548 29266 leader_election.cc:304] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438; no voters: 6e1ec9fa14084d698da867035b48fc13, 6f21cd83f94b4a75ac8648f471aeb85b
I20250903 02:10:21.872680 29915 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 02:10:21.873986 29924 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:21.875430 29639 raft_consensus.cc:1238] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Rejecting Update request from peer 7b012cbd73294b4a869a1515bfd63348 for earlier term 3. Current term is 4. Ops: []
I20250903 02:10:21.875653 29913 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 02:10:21.877091 29924 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 election: Requested vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:10:21.878891 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
I20250903 02:10:21.879140 29927 consensus_queue.cc:1046] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20250903 02:10:21.881955 29603 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.35.193:41955, user_credentials={real_user=slave}} blocked reactor thread for 50889.4us
I20250903 02:10:21.885216 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:10:21.887665 29603 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:21.881672 29914 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:10:21.892722 29930 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250903 02:10:21.893474 29227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "6e1ec9fa14084d698da867035b48fc13" is_pre_election: true
I20250903 02:10:21.902231 29929 raft_consensus.cc:3053] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 3 LEADER]: Stepping down as leader of term 3
I20250903 02:10:21.902655 29929 raft_consensus.cc:738] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:10:21.903450 29929 consensus_queue.cc:260] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:21.904886 29929 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:10:21.904350 29214 catalog_manager.cc:5582] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 reported cstate change: term changed from 3 to 4, leader changed from 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194) to 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195). New cstate: current_term: 4 leader_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: HEALTHY } } }
I20250903 02:10:21.905417 29292 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "36f57bf97db44ab1a86d32e59b4ad438" is_pre_election: true
I20250903 02:10:21.906903 29808 leader_election.cc:304] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f21cd83f94b4a75ac8648f471aeb85b; no voters: 36f57bf97db44ab1a86d32e59b4ad438, 6e1ec9fa14084d698da867035b48fc13
I20250903 02:10:21.916493 29932 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 02:10:21.966107 29482 raft_consensus.cc:1273] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Refusing update from remote peer 1cecbbd1da7046a19f694aa1fe677ed6: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250903 02:10:21.967599 29930 consensus_queue.cc:1035] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250903 02:10:21.986256 29562 raft_consensus.cc:1273] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Refusing update from remote peer 1cecbbd1da7046a19f694aa1fe677ed6: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250903 02:10:21.994796 29923 consensus_queue.cc:1035] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250903 02:10:22.002557 29929 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:10:22.002995 29929 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.005239 29929 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:10:22.005918 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:10:22.006999 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:10:22.008708 29527 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348; no voters: 1cecbbd1da7046a19f694aa1fe677ed6, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:22.017028 29929 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250903 02:10:22.021468 29914 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:10:22.021929 29914 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:22.024703 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:10:22.025104 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:10:22.025364 29639 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 4.
I20250903 02:10:22.025733 29482 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 4.
I20250903 02:10:22.026660 29527 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348; no voters:
I20250903 02:10:22.027451 29929 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250903 02:10:22.027840 29929 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:10:22.028141 29929 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Advancing to term 5
I20250903 02:10:22.023509 29914 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:10:22.038702 29929 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:22.040856 29929 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 election: Requested vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:10:22.041608 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:10:22.041751 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:10:22.042062 29639 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Advancing to term 5
I20250903 02:10:22.042378 29482 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Advancing to term 5
I20250903 02:10:22.046422 29639 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 5.
I20250903 02:10:22.047413 29527 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348; no voters:
I20250903 02:10:22.048072 29929 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Leader election won for term 5
I20250903 02:10:22.048556 29929 raft_consensus.cc:695] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 5 LEADER]: Becoming Leader. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:10:22.050024 29929 consensus_queue.cc:237] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:22.050668 29482 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 5.
I20250903 02:10:22.057497 29214 catalog_manager.cc:5582] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7b012cbd73294b4a869a1515bfd63348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: UNKNOWN } } }
I20250903 02:10:22.095233 29929 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:10:22.096388 29929 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.167547 29930 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:10:22.168233 29930 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.170498 29930 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:10:22.099836 29929 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:10:22.270758 29930 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:10:22.273398 29930 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:22.277323 29930 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
I20250903 02:10:22.277292 29932 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e1ec9fa14084d698da867035b48fc13)
I20250903 02:10:22.278470 29932 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:10:22.280571 29932 leader_election.cc:290] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649)
I20250903 02:10:22.492905 29949 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cecbbd1da7046a19f694aa1fe677ed6)
I20250903 02:10:22.493386 29949 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.495502 29949 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:10:22.497596 29949 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:10:22.498132 29949 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:22.496876 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:10:22.496989 29950 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:22.500809 29949 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
I20250903 02:10:22.500937 29950 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.503717 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:10:22.503125 29950 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:10:22.504370 29639 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c933d23ae9746a8858c7d3ed33c3129 in term 5.
I20250903 02:10:22.505390 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:10:22.506095 29638 raft_consensus.cc:2391] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c933d23ae9746a8858c7d3ed33c3129 in current term 4: Already voted for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in this term.
I20250903 02:10:22.506919 29445 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:10:22.506803 29560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:22.507922 29560 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 4.
W20250903 02:10:22.508244 29603 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.27.35.193:41955 timed out after 0.291s (SENT) [suppressed 11 similar messages]
I20250903 02:10:22.509658 29950 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250903 02:10:22.510234 29950 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:10:22.510681 29950 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Advancing to term 6
I20250903 02:10:22.522650 29914 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1cecbbd1da7046a19f694aa1fe677ed6)
I20250903 02:10:22.522076 29950 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:22.523202 29914 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.526480 29914 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:10:22.526736 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:10:22.527362 29481 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 4.
I20250903 02:10:22.527606 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:10:22.528273 29482 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 3.
I20250903 02:10:22.529237 29950 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 6 election: Requested vote from peers 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:22.529412 29248 debug-util.cc:398] Leaking SignalData structure 0x7b08000f0700 after lost signal to thread 27794
W20250903 02:10:22.530692 29248 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4020 after lost signal to thread 29251
W20250903 02:10:22.531895 29248 debug-util.cc:398] Leaking SignalData structure 0x7b0800150fe0 after lost signal to thread 29316
W20250903 02:10:22.533496 29248 debug-util.cc:398] Leaking SignalData structure 0x7b08000befc0 after lost signal to thread 29508
W20250903 02:10:22.534768 29248 debug-util.cc:398] Leaking SignalData structure 0x7b0800046160 after lost signal to thread 29586
W20250903 02:10:22.535197 29248 debug-util.cc:398] Leaking SignalData structure 0x7b0800097d20 after lost signal to thread 29604
W20250903 02:10:22.544157 29248 debug-util.cc:398] Leaking SignalData structure 0x7b080023f3a0 after lost signal to thread 29664
I20250903 02:10:22.545132 29560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:22.546877 29561 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:22.547492 29561 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c933d23ae9746a8858c7d3ed33c3129 in term 4.
I20250903 02:10:22.548171 29483 raft_consensus.cc:1238] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 6 FOLLOWER]: Rejecting Update request from peer 7b012cbd73294b4a869a1515bfd63348 for earlier term 5. Current term is 6. Ops: []
I20250903 02:10:22.548559 29525 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:10:22.549659 29525 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:10:22.550264 29291 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "36f57bf97db44ab1a86d32e59b4ad438" is_pre_election: true
I20250903 02:10:22.551347 29919 raft_consensus.cc:2802] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250903 02:10:22.551846 29919 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1cecbbd1da7046a19f694aa1fe677ed6)
I20250903 02:10:22.552249 29919 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Advancing to term 5
I20250903 02:10:22.556392 29559 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
I20250903 02:10:22.557649 29227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "6e1ec9fa14084d698da867035b48fc13" is_pre_election: true
I20250903 02:10:22.559114 29808 leader_election.cc:304] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f21cd83f94b4a75ac8648f471aeb85b; no voters: 36f57bf97db44ab1a86d32e59b4ad438, 6e1ec9fa14084d698da867035b48fc13
I20250903 02:10:22.560163 29932 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 02:10:22.561517 29484 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
W20250903 02:10:22.561961 29603 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.35.193:41955, user_credentials={real_user=slave}} blocked reactor thread for 53745.9us
I20250903 02:10:22.562197 29484 raft_consensus.cc:2391] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cecbbd1da7046a19f694aa1fe677ed6 in current term 6: Already voted for candidate 9c933d23ae9746a8858c7d3ed33c3129 in this term.
I20250903 02:10:22.565892 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:22.567886 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:10:22.568425 29481 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cecbbd1da7046a19f694aa1fe677ed6 in term 3.
I20250903 02:10:22.570029 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:10:22.570549 29638 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Advancing to term 6
I20250903 02:10:22.571491 29914 raft_consensus.cc:2802] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250903 02:10:22.571938 29914 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:10:22.572340 29914 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 4 FOLLOWER]: Advancing to term 5
I20250903 02:10:22.573078 29444 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348
I20250903 02:10:22.574298 29444 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6
I20250903 02:10:22.576936 29913 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250903 02:10:22.577548 29913 raft_consensus.cc:2802] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250903 02:10:22.577677 29638 raft_consensus.cc:2466] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c933d23ae9746a8858c7d3ed33c3129 in term 6.
I20250903 02:10:22.577908 29913 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:22.578385 29913 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 3 FOLLOWER]: Advancing to term 4
I20250903 02:10:22.579277 29914 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.581981 29482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:10:22.583469 29914 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 election: Requested vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:10:22.584189 29914 consensus_queue.cc:1046] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250903 02:10:22.585382 29929 raft_consensus.cc:3053] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 5 LEADER]: Stepping down as leader of term 5
I20250903 02:10:22.586300 29929 raft_consensus.cc:738] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:10:22.587049 29929 consensus_queue.cc:260] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:22.590617 29915 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e1ec9fa14084d698da867035b48fc13)
I20250903 02:10:22.588413 29929 raft_consensus.cc:3058] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Advancing to term 6
I20250903 02:10:22.593824 29445 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:10:22.611236 29950 raft_consensus.cc:2802] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 6 FOLLOWER]: Leader election won for term 6
I20250903 02:10:22.591161 29915 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:10:22.612391 29950 raft_consensus.cc:695] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 6 LEADER]: Becoming Leader. State: Replica: 9c933d23ae9746a8858c7d3ed33c3129, State: Running, Role: LEADER
I20250903 02:10:22.613780 29915 leader_election.cc:290] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 6f21cd83f94b4a75ac8648f471aeb85b (127.27.35.252:45587)
I20250903 02:10:22.613401 29950 consensus_queue.cc:237] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:22.619907 29227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36f57bf97db44ab1a86d32e59b4ad438" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "6e1ec9fa14084d698da867035b48fc13" is_pre_election: true
I20250903 02:10:22.623838 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db760726eb304ef0b69159db4bbaef75" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
W20250903 02:10:22.620431 29528 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250903 02:10:22.624938 29836 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "36f57bf97db44ab1a86d32e59b4ad438" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" is_pre_election: true
I20250903 02:10:22.626330 29266 leader_election.cc:304] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36f57bf97db44ab1a86d32e59b4ad438; no voters: 6e1ec9fa14084d698da867035b48fc13, 6f21cd83f94b4a75ac8648f471aeb85b
I20250903 02:10:22.626410 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:10:22.627272 29638 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 4.
I20250903 02:10:22.589291 29913 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.628175 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" is_pre_election: true
I20250903 02:10:22.632567 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:10:22.633160 29638 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Advancing to term 5
I20250903 02:10:22.635178 29559 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348" is_pre_election: true
I20250903 02:10:22.636054 29559 raft_consensus.cc:2391] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cecbbd1da7046a19f694aa1fe677ed6 in current term 5: Already voted for candidate 7b012cbd73294b4a869a1515bfd63348 in this term.
I20250903 02:10:22.636689 29482 raft_consensus.cc:3058] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Advancing to term 5
I20250903 02:10:22.636831 29913 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 4 election: Requested vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:10:22.642658 29482 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 5.
I20250903 02:10:22.643846 29639 raft_consensus.cc:1238] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Rejecting Update request from peer 7b012cbd73294b4a869a1515bfd63348 for earlier term 5. Current term is 6. Ops: []
I20250903 02:10:22.628166 29915 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 02:10:22.647197 29639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:10:22.652335 29525 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129; no voters:
I20250903 02:10:22.652876 29562 raft_consensus.cc:2408] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cecbbd1da7046a19f694aa1fe677ed6 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250903 02:10:22.653409 29929 raft_consensus.cc:2802] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Leader election won for term 5
I20250903 02:10:22.653952 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129" is_pre_election: true
I20250903 02:10:22.654678 29481 raft_consensus.cc:2391] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cecbbd1da7046a19f694aa1fe677ed6 in current term 5: Already voted for candidate 7b012cbd73294b4a869a1515bfd63348 in this term.
I20250903 02:10:22.655066 29604 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:22.655131 29929 raft_consensus.cc:695] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 5 LEADER]: Becoming Leader. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:10:22.655920 29930 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 02:10:22.656332 29929 consensus_queue.cc:237] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.659538 29638 raft_consensus.cc:2466] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 5.
I20250903 02:10:22.661589 29562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2da3487154804c71a24242048c1b0c16" candidate_uuid: "9c933d23ae9746a8858c7d3ed33c3129" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "7b012cbd73294b4a869a1515bfd63348"
I20250903 02:10:22.662609 29603 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:22.662076 29919 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.663851 29444 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c933d23ae9746a8858c7d3ed33c3129; no voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348
I20250903 02:10:22.664438 29559 raft_consensus.cc:1238] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Rejecting Update request from peer 1cecbbd1da7046a19f694aa1fe677ed6 for earlier term 4. Current term is 5. Ops: []
I20250903 02:10:22.665752 29913 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250903 02:10:22.669905 29930 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250903 02:10:22.677955 29907 consensus_queue.cc:1046] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20250903 02:10:22.681265 29481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "9c933d23ae9746a8858c7d3ed33c3129"
I20250903 02:10:22.681794 29928 raft_consensus.cc:3053] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 LEADER]: Stepping down as leader of term 4
I20250903 02:10:22.682271 29928 raft_consensus.cc:738] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1cecbbd1da7046a19f694aa1fe677ed6, State: Running, Role: LEADER
I20250903 02:10:22.685410 29214 catalog_manager.cc:5582] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "7b012cbd73294b4a869a1515bfd63348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: UNKNOWN } } }
I20250903 02:10:22.687479 29919 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 election: Requested vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195:45481)
I20250903 02:10:22.691052 29215 catalog_manager.cc:5582] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 reported cstate change: term changed from 5 to 6, leader changed from 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194) to 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193). New cstate: current_term: 6 leader_uuid: "9c933d23ae9746a8858c7d3ed33c3129" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: HEALTHY } } }
I20250903 02:10:22.683122 29928 consensus_queue.cc:260] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.695307 29928 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 4 FOLLOWER]: Advancing to term 5
I20250903 02:10:22.696273 29638 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "beaf5654ab144e23a627269917ad5ee9" candidate_uuid: "7b012cbd73294b4a869a1515bfd63348" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1cecbbd1da7046a19f694aa1fe677ed6"
I20250903 02:10:22.710352 29638 raft_consensus.cc:2466] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b012cbd73294b4a869a1515bfd63348 in term 5.
I20250903 02:10:22.712556 29527 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6, 7b012cbd73294b4a869a1515bfd63348; no voters: 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:22.714382 29919 raft_consensus.cc:2802] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Leader election won for term 5
I20250903 02:10:22.714881 29919 raft_consensus.cc:695] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 5 LEADER]: Becoming Leader. State: Replica: 7b012cbd73294b4a869a1515bfd63348, State: Running, Role: LEADER
I20250903 02:10:22.715692 29919 consensus_queue.cc:237] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:22.724927 29214 catalog_manager.cc:5582] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 reported cstate change: term changed from 4 to 5, leader changed from 1cecbbd1da7046a19f694aa1fe677ed6 (127.27.35.195) to 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194). New cstate: current_term: 5 leader_uuid: "7b012cbd73294b4a869a1515bfd63348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } health_report { overall_health: UNKNOWN } } }
I20250903 02:10:22.749147 29638 raft_consensus.cc:1273] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Refusing update from remote peer 9c933d23ae9746a8858c7d3ed33c3129: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20250903 02:10:22.751398 29950 consensus_queue.cc:1035] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250903 02:10:22.765341 29559 raft_consensus.cc:1273] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 6 FOLLOWER]: Refusing update from remote peer 9c933d23ae9746a8858c7d3ed33c3129: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20250903 02:10:22.765769 29559 pending_rounds.cc:85] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348: Aborting all ops after (but not including) 5
I20250903 02:10:22.766021 29559 pending_rounds.cc:107] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20250903 02:10:22.766286 29559 raft_consensus.cc:2887] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250903 02:10:22.766722 29639 raft_consensus.cc:1273] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Refusing update from remote peer 7b012cbd73294b4a869a1515bfd63348: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250903 02:10:22.768054 29919 consensus_queue.cc:1035] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250903 02:10:22.768293 29950 consensus_queue.cc:1035] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250903 02:10:22.793145 29481 raft_consensus.cc:1273] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Refusing update from remote peer 7b012cbd73294b4a869a1515bfd63348: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250903 02:10:22.794914 29914 consensus_queue.cc:1035] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250903 02:10:22.801528 29640 raft_consensus.cc:1273] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Refusing update from remote peer 7b012cbd73294b4a869a1515bfd63348: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250903 02:10:22.804522 29914 consensus_queue.cc:1035] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250903 02:10:22.814729 29482 raft_consensus.cc:3058] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 4 FOLLOWER]: Advancing to term 5
I20250903 02:10:22.827507 29482 raft_consensus.cc:1273] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Refusing update from remote peer 7b012cbd73294b4a869a1515bfd63348: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250903 02:10:22.828910 29914 consensus_queue.cc:1035] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250903 02:10:23.397756 29583 debug-util.cc:398] Leaking SignalData structure 0x7b08001cf060 after lost signal to thread 27794
W20250903 02:10:23.398785 29583 debug-util.cc:398] Leaking SignalData structure 0x7b08001cf7c0 after lost signal to thread 29251
W20250903 02:10:23.399539 29583 debug-util.cc:398] Leaking SignalData structure 0x7b08001cfe80 after lost signal to thread 29316
W20250903 02:10:23.400610 29583 debug-util.cc:398] Leaking SignalData structure 0x7b08000dbe20 after lost signal to thread 29508
W20250903 02:10:23.401640 29583 debug-util.cc:398] Leaking SignalData structure 0x7b080016aec0 after lost signal to thread 29586
W20250903 02:10:23.402611 29583 debug-util.cc:398] Leaking SignalData structure 0x7b0800221e80 after lost signal to thread 29664
W20250903 02:10:23.403607 29583 debug-util.cc:398] Leaking SignalData structure 0x7b080024fd80 after lost signal to thread 29861
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:784: Failure
Expected equality of these values:
live_row_count
Which is: 67
table_info->GetMetrics()->live_row_count->value()
Which is: 134
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:768: Failure
Expected: check_function(table_infos[0].get(), live_row_count) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:786: Failure
Expected: GetLeaderMasterAndRun(live_row_count, [&] ( TableInfo* table_info, int64_t live_row_count) { do { AssertEventually([&] () { switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(table_info->GetMetrics()->TableSupportsLiveRowCount())) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 783, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "table_info->GetMetrics()->TableSupportsLiveRowCount()", "false", "true") .c_str()) = ::testing::Message(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("live_row_count", "table_info->GetMetrics()->live_row_count->value()", live_row_count, table_info->GetMetrics()->live_row_count->value()))) ; else return ::testing::internal::AssertHelper(::testing::TestPartResult::kFatalFailure, "/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc", 784, gtest_ar.failure_message()) = ::testing::Message(); }); do { if (testing::Test::HasFatalFailure()) { return; } } while (0); } while (0); }) doesn't generate new fatal failures in the current thread.
Actual: it does.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/ts_tablet_manager-itest.cc:896: Failure
Expected: CheckStats(kRowsCount) doesn't generate new fatal failures in the current thread.
Actual: it does.
I20250903 02:10:24.776181 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
W20250903 02:10:24.797681 29525 consensus_peers.cc:489] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 -> Peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Couldn't send request to peer 9c933d23ae9746a8858c7d3ed33c3129. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250903 02:10:24.838366 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:24.839370 27791 tablet_replica.cc:331] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129: stopping tablet replica
W20250903 02:10:24.839448 29525 consensus_peers.cc:489] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 -> Peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Couldn't send request to peer 9c933d23ae9746a8858c7d3ed33c3129. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250903 02:10:24.840620 27791 raft_consensus.cc:2241] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:24.841935 27791 raft_consensus.cc:2270] T beaf5654ab144e23a627269917ad5ee9 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:24.845126 27791 tablet_replica.cc:331] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129: stopping tablet replica
I20250903 02:10:24.845872 27791 raft_consensus.cc:2241] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:24.846433 27791 raft_consensus.cc:2270] T 2da3487154804c71a24242048c1b0c16 P 9c933d23ae9746a8858c7d3ed33c3129 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:24.848948 27791 tablet_replica.cc:331] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129: stopping tablet replica
I20250903 02:10:24.849673 27791 raft_consensus.cc:2241] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 6 LEADER]: Raft consensus shutting down.
I20250903 02:10:24.850589 27791 raft_consensus.cc:2270] T db760726eb304ef0b69159db4bbaef75 P 9c933d23ae9746a8858c7d3ed33c3129 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:24.879490 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:10:24.906065 27791 tablet_server.cc:178] TabletServer@127.27.35.194:0 shutting down...
I20250903 02:10:24.957962 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:24.965354 27791 tablet_replica.cc:331] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348: stopping tablet replica
I20250903 02:10:24.965894 27791 raft_consensus.cc:2241] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 5 LEADER]: Raft consensus shutting down.
I20250903 02:10:24.966727 27791 raft_consensus.cc:2270] T 2da3487154804c71a24242048c1b0c16 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:24.968818 27791 tablet_replica.cc:331] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348: stopping tablet replica
I20250903 02:10:24.969275 27791 raft_consensus.cc:2241] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:24.969662 27791 raft_consensus.cc:2270] T db760726eb304ef0b69159db4bbaef75 P 7b012cbd73294b4a869a1515bfd63348 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:24.971551 27791 tablet_replica.cc:331] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348: stopping tablet replica
I20250903 02:10:24.972021 27791 raft_consensus.cc:2241] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 5 LEADER]: Raft consensus shutting down.
I20250903 02:10:24.972829 27791 raft_consensus.cc:2270] T beaf5654ab144e23a627269917ad5ee9 P 7b012cbd73294b4a869a1515bfd63348 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:25.032676 29975 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c933d23ae9746a8858c7d3ed33c3129)
I20250903 02:10:25.033388 29975 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:25.041891 29975 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
W20250903 02:10:25.059019 29603 leader_election.cc:336] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:25.059725 29604 leader_election.cc:336] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:25.060252 29604 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.061097 29975 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 02:10:25.131011 29975 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:25.131611 29975 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:25.133761 29975 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:25.139133 29603 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:25.139825 29604 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:25.140322 29604 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.141106 29975 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 02:10:25.164539 29975 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:25.165360 29975 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:25.169807 29975 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:25.174181 29604 leader_election.cc:336] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
W20250903 02:10:25.174749 29603 leader_election.cc:336] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
I20250903 02:10:25.175215 29603 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.176020 29975 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 02:10:25.334486 29975 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c933d23ae9746a8858c7d3ed33c3129)
I20250903 02:10:25.335201 29975 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:25.337397 29975 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
W20250903 02:10:25.345986 29603 leader_election.cc:336] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:25.346626 29604 leader_election.cc:336] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:25.347100 29604 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.347911 29975 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 02:10:25.362126 29975 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:25.362787 29975 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:25.365178 29975 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:25.370226 29603 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:25.370915 29604 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:25.371393 29604 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.372216 29975 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 02:10:25.392889 29975 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:25.393529 29975 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:25.395710 29975 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:25.403561 29603 leader_election.cc:336] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:25.404242 29604 leader_election.cc:336] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:25.404752 29604 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.405949 29975 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 02:10:25.555889 29975 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c933d23ae9746a8858c7d3ed33c3129)
I20250903 02:10:25.556552 29975 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:25.559192 29975 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
W20250903 02:10:25.563977 29604 leader_election.cc:336] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
W20250903 02:10:25.564813 29603 leader_election.cc:336] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
I20250903 02:10:25.565270 29603 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.566016 29975 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 02:10:25.719270 29975 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:25.719970 29975 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:25.723253 29975 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:25.730515 29604 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
W20250903 02:10:25.733110 29603 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
I20250903 02:10:25.733716 29603 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.734762 29975 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 02:10:25.746443 29975 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:25.747063 29975 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:25.749411 29975 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:25.754295 29603 leader_election.cc:336] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:25.755383 29604 leader_election.cc:336] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:25.755930 29604 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.756848 29975 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 02:10:25.903303 29975 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c933d23ae9746a8858c7d3ed33c3129)
I20250903 02:10:25.903968 29975 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:25.906100 29975 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
W20250903 02:10:25.911661 29604 leader_election.cc:336] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
W20250903 02:10:25.913640 29603 leader_election.cc:336] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
I20250903 02:10:25.914127 29603 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.914898 29975 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 02:10:25.966815 29975 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:25.967414 29975 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:25.969799 29975 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:25.974340 29603 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:25.976272 29604 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:25.976938 29604 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:25.978137 29975 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250903 02:10:25.995865 27791 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 29583)
I20250903 02:10:26.013206 29975 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:26.013947 29975 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:26.017149 29975 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:26.026088 29604 leader_election.cc:336] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
W20250903 02:10:26.027055 29603 leader_election.cc:336] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
I20250903 02:10:26.027539 29603 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:26.028307 29975 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 02:10:26.337455 29975 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:26.337960 29975 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:26.339561 29975 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:26.343195 29603 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:26.343766 29604 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:26.344210 29604 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:26.344928 29975 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 02:10:26.724336 29975 raft_consensus.cc:491] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 9c933d23ae9746a8858c7d3ed33c3129)
I20250903 02:10:26.724943 29975 raft_consensus.cc:513] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } }
I20250903 02:10:26.727088 29975 leader_election.cc:290] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673), 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955)
W20250903 02:10:26.730966 29603 leader_election.cc:336] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:26.731434 29604 leader_election.cc:336] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:26.731784 29604 leader_election.cc:304] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:26.732345 29975 raft_consensus.cc:2747] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250903 02:10:26.811664 29975 raft_consensus.cc:491] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:26.812229 29975 raft_consensus.cc:513] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:26.814921 29975 leader_election.cc:290] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:26.818219 29603 leader_election.cc:336] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:26.819655 29604 leader_election.cc:336] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:26.820073 29604 leader_election.cc:304] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:26.820653 29975 raft_consensus.cc:2747] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250903 02:10:26.996747 27791 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 29583)
I20250903 02:10:27.055419 27791 tablet_server.cc:195] TabletServer@127.27.35.194:0 shutdown complete.
I20250903 02:10:27.073165 27791 tablet_server.cc:178] TabletServer@127.27.35.195:0 shutting down...
I20250903 02:10:27.078641 29975 raft_consensus.cc:491] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7b012cbd73294b4a869a1515bfd63348)
I20250903 02:10:27.081439 29975 raft_consensus.cc:513] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c933d23ae9746a8858c7d3ed33c3129" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 41955 } } peers { permanent_uuid: "7b012cbd73294b4a869a1515bfd63348" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 42673 } } peers { permanent_uuid: "1cecbbd1da7046a19f694aa1fe677ed6" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 45481 } }
I20250903 02:10:27.084237 29975 leader_election.cc:290] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955), 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673)
W20250903 02:10:27.090691 29603 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 9c933d23ae9746a8858c7d3ed33c3129 (127.27.35.193:41955): Network error: Client connection negotiation failed: client connection to 127.27.35.193:41955: connect: Connection refused (error 111)
W20250903 02:10:27.094305 29604 leader_election.cc:336] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7b012cbd73294b4a869a1515bfd63348 (127.27.35.194:42673): Network error: Client connection negotiation failed: client connection to 127.27.35.194:42673: connect: Connection refused (error 111)
I20250903 02:10:27.094695 29604 leader_election.cc:304] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cecbbd1da7046a19f694aa1fe677ed6; no voters: 7b012cbd73294b4a869a1515bfd63348, 9c933d23ae9746a8858c7d3ed33c3129
I20250903 02:10:27.098048 29975 raft_consensus.cc:2747] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250903 02:10:27.116168 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:27.116938 27791 tablet_replica.cc:331] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6: stopping tablet replica
I20250903 02:10:27.117765 27791 raft_consensus.cc:2241] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:27.118347 27791 raft_consensus.cc:2270] T db760726eb304ef0b69159db4bbaef75 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:27.120563 27791 tablet_replica.cc:331] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6: stopping tablet replica
I20250903 02:10:27.120991 27791 raft_consensus.cc:2241] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:27.121466 27791 raft_consensus.cc:2270] T 2da3487154804c71a24242048c1b0c16 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:27.123412 27791 tablet_replica.cc:331] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6: stopping tablet replica
I20250903 02:10:27.123927 27791 raft_consensus.cc:2241] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:27.124373 27791 raft_consensus.cc:2270] T beaf5654ab144e23a627269917ad5ee9 P 1cecbbd1da7046a19f694aa1fe677ed6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:27.147039 27791 tablet_server.cc:195] TabletServer@127.27.35.195:0 shutdown complete.
I20250903 02:10:27.162757 27791 master.cc:561] Master@127.27.35.254:41877 shutting down...
I20250903 02:10:27.185276 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 3 LEADER]: Raft consensus shutting down.
I20250903 02:10:27.186204 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:27.186585 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6e1ec9fa14084d698da867035b48fc13: stopping tablet replica
I20250903 02:10:27.227257 27791 master.cc:583] Master@127.27.35.254:41877 shutdown complete.
I20250903 02:10:27.249145 27791 master.cc:561] Master@127.27.35.253:42649 shutting down...
I20250903 02:10:27.273375 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:27.274194 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:27.274559 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P 36f57bf97db44ab1a86d32e59b4ad438: stopping tablet replica
I20250903 02:10:27.307093 29982 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e1ec9fa14084d698da867035b48fc13)
I20250903 02:10:27.307657 29982 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1ec9fa14084d698da867035b48fc13" member_type: VOTER last_known_addr { host: "127.27.35.254" port: 41877 } } peers { permanent_uuid: "36f57bf97db44ab1a86d32e59b4ad438" member_type: VOTER last_known_addr { host: "127.27.35.253" port: 42649 } } peers { permanent_uuid: "6f21cd83f94b4a75ac8648f471aeb85b" member_type: VOTER last_known_addr { host: "127.27.35.252" port: 45587 } }
I20250903 02:10:27.309703 29982 leader_election.cc:290] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877), 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649)
W20250903 02:10:27.316376 29808 leader_election.cc:336] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6e1ec9fa14084d698da867035b48fc13 (127.27.35.254:41877): Network error: Client connection negotiation failed: client connection to 127.27.35.254:41877: connect: Connection refused (error 111)
W20250903 02:10:27.317035 29808 leader_election.cc:336] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 36f57bf97db44ab1a86d32e59b4ad438 (127.27.35.253:42649): Network error: Client connection negotiation failed: client connection to 127.27.35.253:42649: connect: Connection refused (error 111)
I20250903 02:10:27.317405 29808 leader_election.cc:304] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f21cd83f94b4a75ac8648f471aeb85b; no voters: 36f57bf97db44ab1a86d32e59b4ad438, 6e1ec9fa14084d698da867035b48fc13
I20250903 02:10:27.318164 29982 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250903 02:10:27.323024 27791 master.cc:583] Master@127.27.35.253:42649 shutdown complete.
I20250903 02:10:27.339375 27791 master.cc:561] Master@127.27.35.252:45587 shutting down...
I20250903 02:10:27.365990 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:27.366710 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6f21cd83f94b4a75ac8648f471aeb85b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:27.367017 27791 tablet_replica.cc:331] stopping tablet replica
I20250903 02:10:27.387826 27791 master.cc:583] Master@127.27.35.252:45587 shutdown complete.
I20250903 02:10:27.415393 27791 test_util.cc:183] -----------------------------------------------
I20250903 02:10:27.415624 27791 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestTableStats.1756865376507568-27791-0
[ FAILED ] TsTabletManagerITest.TestTableStats (37456 ms)
[ RUN ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy
I20250903 02:10:27.422415 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:43777
I20250903 02:10:27.423583 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:27.428997 29990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:27.429751 29991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:27.432312 27791 server_base.cc:1047] running on GCE node
W20250903 02:10:27.432355 29993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:27.433449 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:27.433674 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:27.433827 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865427433810 us; error 0 us; skew 500 ppm
I20250903 02:10:27.434340 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:27.436774 27791 webserver.cc:480] Webserver started at http://127.27.35.254:33165/ using document root <none> and password file <none>
I20250903 02:10:27.437280 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:27.437467 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:27.437708 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:27.438951 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "6d380471611f47469d70ee751f7d7398"
format_stamp: "Formatted at 2025-09-03 02:10:27 on dist-test-slave-h78g"
I20250903 02:10:27.443456 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250903 02:10:27.446852 29998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:27.447790 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:10:27.448042 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "6d380471611f47469d70ee751f7d7398"
format_stamp: "Formatted at 2025-09-03 02:10:27 on dist-test-slave-h78g"
I20250903 02:10:27.448287 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:27.460453 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:27.461643 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:27.501555 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:43777
I20250903 02:10:27.501684 30049 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:43777 every 8 connection(s)
I20250903 02:10:27.506076 30050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:27.517771 30050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398: Bootstrap starting.
I20250903 02:10:27.522956 30050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:27.527362 30050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398: No bootstrap required, opened a new log
I20250903 02:10:27.529613 30050 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d380471611f47469d70ee751f7d7398" member_type: VOTER }
I20250903 02:10:27.530001 30050 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:27.530251 30050 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d380471611f47469d70ee751f7d7398, State: Initialized, Role: FOLLOWER
I20250903 02:10:27.530843 30050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d380471611f47469d70ee751f7d7398" member_type: VOTER }
I20250903 02:10:27.531292 30050 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:27.531518 30050 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:27.531790 30050 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:27.536147 30050 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d380471611f47469d70ee751f7d7398" member_type: VOTER }
I20250903 02:10:27.536680 30050 leader_election.cc:304] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d380471611f47469d70ee751f7d7398; no voters:
I20250903 02:10:27.537813 30050 leader_election.cc:290] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:27.538214 30053 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:27.539582 30053 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 1 LEADER]: Becoming Leader. State: Replica: 6d380471611f47469d70ee751f7d7398, State: Running, Role: LEADER
I20250903 02:10:27.540304 30053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d380471611f47469d70ee751f7d7398" member_type: VOTER }
I20250903 02:10:27.540846 30050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:10:27.546437 30054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d380471611f47469d70ee751f7d7398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d380471611f47469d70ee751f7d7398" member_type: VOTER } }
I20250903 02:10:27.547112 30054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:27.548658 30055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d380471611f47469d70ee751f7d7398. Latest consensus state: current_term: 1 leader_uuid: "6d380471611f47469d70ee751f7d7398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d380471611f47469d70ee751f7d7398" member_type: VOTER } }
I20250903 02:10:27.549317 30061 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:10:27.549340 30055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:27.554615 30061 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:10:27.557422 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:10:27.563971 30061 catalog_manager.cc:1349] Generated new cluster ID: f952a2dcc50249b585226a04050e0723
I20250903 02:10:27.564244 30061 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:10:27.580740 30061 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:10:27.581983 30061 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:10:27.594161 30061 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398: Generated new TSK 0
I20250903 02:10:27.594748 30061 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:10:27.623912 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:27.630122 30071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:27.631376 30072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:27.633594 30074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:27.634807 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:27.635915 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:27.636108 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:27.636265 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865427636247 us; error 0 us; skew 500 ppm
I20250903 02:10:27.636811 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:27.639000 27791 webserver.cc:480] Webserver started at http://127.27.35.193:42831/ using document root <none> and password file <none>
I20250903 02:10:27.639511 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:27.639694 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:27.640003 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:27.641146 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "3f887198f79344209034c9ddf12fe026"
format_stamp: "Formatted at 2025-09-03 02:10:27 on dist-test-slave-h78g"
I20250903 02:10:27.645531 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:10:27.648648 30079 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:27.649341 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:10:27.649611 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "3f887198f79344209034c9ddf12fe026"
format_stamp: "Formatted at 2025-09-03 02:10:27 on dist-test-slave-h78g"
I20250903 02:10:27.649873 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:27.663671 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:27.664999 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:27.666460 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:10:27.668720 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:27.668931 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:27.669184 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:27.669371 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:27.712241 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:35771
I20250903 02:10:27.712383 30141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:35771 every 8 connection(s)
I20250903 02:10:27.717450 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:27.725677 30146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:27.727418 30147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:27.729224 30149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:27.730501 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:27.731400 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:27.731607 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:27.731736 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865427731724 us; error 0 us; skew 500 ppm
I20250903 02:10:27.732265 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:27.735088 27791 webserver.cc:480] Webserver started at http://127.27.35.194:34633/ using document root <none> and password file <none>
I20250903 02:10:27.735839 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:27.736102 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:27.736383 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:27.736716 30142 heartbeater.cc:344] Connected to a master server at 127.27.35.254:43777
I20250903 02:10:27.737056 30142 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:27.737663 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-1-root/instance:
uuid: "591ccbc74880444a9b3bbf124e84e6d3"
format_stamp: "Formatted at 2025-09-03 02:10:27 on dist-test-slave-h78g"
I20250903 02:10:27.737728 30142 heartbeater.cc:507] Master 127.27.35.254:43777 requested a full tablet report, sending...
I20250903 02:10:27.739916 30015 ts_manager.cc:194] Registered new tserver with Master: 3f887198f79344209034c9ddf12fe026 (127.27.35.193:35771)
I20250903 02:10:27.741642 30015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60718
I20250903 02:10:27.742877 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250903 02:10:27.746742 30154 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:27.747562 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 02:10:27.748039 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-1-root
uuid: "591ccbc74880444a9b3bbf124e84e6d3"
format_stamp: "Formatted at 2025-09-03 02:10:27 on dist-test-slave-h78g"
I20250903 02:10:27.748360 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:27.769057 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:27.770270 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:27.771837 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:10:27.774072 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:27.774286 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:27.774549 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:27.774705 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:27.817032 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.194:40951
I20250903 02:10:27.817144 30216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.194:40951 every 8 connection(s)
I20250903 02:10:27.822121 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:27.828943 30220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:27.830762 30221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:27.832337 30223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:27.832903 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:27.833874 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250903 02:10:27.833986 30217 heartbeater.cc:344] Connected to a master server at 127.27.35.254:43777
W20250903 02:10:27.834174 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:27.834412 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865427834393 us; error 0 us; skew 500 ppm
I20250903 02:10:27.834465 30217 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:27.834992 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:27.835274 30217 heartbeater.cc:507] Master 127.27.35.254:43777 requested a full tablet report, sending...
I20250903 02:10:27.837404 30015 ts_manager.cc:194] Registered new tserver with Master: 591ccbc74880444a9b3bbf124e84e6d3 (127.27.35.194:40951)
I20250903 02:10:27.837716 27791 webserver.cc:480] Webserver started at http://127.27.35.195:33487/ using document root <none> and password file <none>
I20250903 02:10:27.838269 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:27.838447 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:27.838717 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:27.838969 30015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60724
I20250903 02:10:27.839886 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-2-root/instance:
uuid: "1fec8835051845efbf4798a25ce86527"
format_stamp: "Formatted at 2025-09-03 02:10:27 on dist-test-slave-h78g"
I20250903 02:10:27.844875 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250903 02:10:27.848600 30228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:27.849529 27791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250903 02:10:27.849881 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-2-root
uuid: "1fec8835051845efbf4798a25ce86527"
format_stamp: "Formatted at 2025-09-03 02:10:27 on dist-test-slave-h78g"
I20250903 02:10:27.850173 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:27.861790 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:27.863040 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:27.864610 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:10:27.867077 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:27.867305 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:27.867560 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:27.867719 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:27.908583 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.195:41659
I20250903 02:10:27.908686 30290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.195:41659 every 8 connection(s)
I20250903 02:10:27.922585 30291 heartbeater.cc:344] Connected to a master server at 127.27.35.254:43777
I20250903 02:10:27.922966 30291 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:27.923682 30291 heartbeater.cc:507] Master 127.27.35.254:43777 requested a full tablet report, sending...
I20250903 02:10:27.925660 30015 ts_manager.cc:194] Registered new tserver with Master: 1fec8835051845efbf4798a25ce86527 (127.27.35.195:41659)
I20250903 02:10:27.926625 27791 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014660787s
I20250903 02:10:27.926992 30015 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60728
I20250903 02:10:27.959875 30015 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20250903 02:10:27.961383 30015 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 02:10:27.999558 30182 tablet_service.cc:1468] Processing CreateTablet for tablet 49f005132e9e40809a663f747b506d19 (DEFAULT_TABLE table=test-table [id=ba3173b9fc3947269a9874ea261ae95f]), partition=
I20250903 02:10:28.000813 30182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f005132e9e40809a663f747b506d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:28.005146 30107 tablet_service.cc:1468] Processing CreateTablet for tablet 49f005132e9e40809a663f747b506d19 (DEFAULT_TABLE table=test-table [id=ba3173b9fc3947269a9874ea261ae95f]), partition=
I20250903 02:10:28.007345 30107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f005132e9e40809a663f747b506d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:28.012722 30256 tablet_service.cc:1468] Processing CreateTablet for tablet 49f005132e9e40809a663f747b506d19 (DEFAULT_TABLE table=test-table [id=ba3173b9fc3947269a9874ea261ae95f]), partition=
I20250903 02:10:28.013656 30256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f005132e9e40809a663f747b506d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:28.020769 30307 tablet_bootstrap.cc:492] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Bootstrap starting.
I20250903 02:10:28.028088 30307 tablet_bootstrap.cc:654] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:28.029484 30308 tablet_bootstrap.cc:492] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: Bootstrap starting.
I20250903 02:10:28.031325 30309 tablet_bootstrap.cc:492] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527: Bootstrap starting.
I20250903 02:10:28.034118 30307 tablet_bootstrap.cc:492] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: No bootstrap required, opened a new log
I20250903 02:10:28.034574 30307 ts_tablet_manager.cc:1397] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20250903 02:10:28.037039 30307 raft_consensus.cc:357] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.036980 30308 tablet_bootstrap.cc:654] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:28.037672 30307 raft_consensus.cc:383] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:28.038280 30307 raft_consensus.cc:738] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591ccbc74880444a9b3bbf124e84e6d3, State: Initialized, Role: FOLLOWER
I20250903 02:10:28.038975 30309 tablet_bootstrap.cc:654] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:28.039191 30307 consensus_queue.cc:260] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.042575 30217 heartbeater.cc:499] Master 127.27.35.254:43777 was elected leader, sending a full tablet report...
I20250903 02:10:28.043195 30307 ts_tablet_manager.cc:1428] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Time spent starting tablet: real 0.008s user 0.007s sys 0.002s
W20250903 02:10:28.043596 30307 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20250903 02:10:28.043926 30308 tablet_bootstrap.cc:492] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: No bootstrap required, opened a new log
I20250903 02:10:28.044394 30308 ts_tablet_manager.cc:1397] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20250903 02:10:28.045111 30309 tablet_bootstrap.cc:492] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527: No bootstrap required, opened a new log
I20250903 02:10:28.045573 30309 ts_tablet_manager.cc:1397] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250903 02:10:28.046689 30308 raft_consensus.cc:357] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.047401 30308 raft_consensus.cc:383] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:28.047638 30308 raft_consensus.cc:738] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f887198f79344209034c9ddf12fe026, State: Initialized, Role: FOLLOWER
I20250903 02:10:28.048547 30309 raft_consensus.cc:357] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.048712 30308 consensus_queue.cc:260] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.049430 30309 raft_consensus.cc:383] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:28.050299 30309 raft_consensus.cc:738] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fec8835051845efbf4798a25ce86527, State: Initialized, Role: FOLLOWER
I20250903 02:10:28.050915 30309 consensus_queue.cc:260] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.052608 30142 heartbeater.cc:499] Master 127.27.35.254:43777 was elected leader, sending a full tablet report...
I20250903 02:10:28.053270 30308 ts_tablet_manager.cc:1428] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
W20250903 02:10:28.053566 30308 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20250903 02:10:28.053927 30309 ts_tablet_manager.cc:1428] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20250903 02:10:28.054241 30309 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20250903 02:10:28.054363 30291 heartbeater.cc:499] Master 127.27.35.254:43777 was elected leader, sending a full tablet report...
W20250903 02:10:28.070963 30218 tablet.cc:2378] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250903 02:10:28.334234 30314 raft_consensus.cc:491] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:10:28.334719 30314 raft_consensus.cc:513] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.336730 30314 leader_election.cc:290] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 591ccbc74880444a9b3bbf124e84e6d3 (127.27.35.194:40951), 1fec8835051845efbf4798a25ce86527 (127.27.35.195:41659)
I20250903 02:10:28.346812 30192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49f005132e9e40809a663f747b506d19" candidate_uuid: "3f887198f79344209034c9ddf12fe026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591ccbc74880444a9b3bbf124e84e6d3" is_pre_election: true
I20250903 02:10:28.346966 30266 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49f005132e9e40809a663f747b506d19" candidate_uuid: "3f887198f79344209034c9ddf12fe026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fec8835051845efbf4798a25ce86527" is_pre_election: true
I20250903 02:10:28.347452 30192 raft_consensus.cc:2466] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f887198f79344209034c9ddf12fe026 in term 0.
I20250903 02:10:28.347641 30266 raft_consensus.cc:2466] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f887198f79344209034c9ddf12fe026 in term 0.
I20250903 02:10:28.348598 30080 leader_election.cc:304] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f887198f79344209034c9ddf12fe026, 591ccbc74880444a9b3bbf124e84e6d3; no voters:
I20250903 02:10:28.349258 30314 raft_consensus.cc:2802] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 02:10:28.349560 30314 raft_consensus.cc:491] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:10:28.349817 30314 raft_consensus.cc:3058] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:28.355089 30314 raft_consensus.cc:513] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.356732 30314 leader_election.cc:290] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [CANDIDATE]: Term 1 election: Requested vote from peers 591ccbc74880444a9b3bbf124e84e6d3 (127.27.35.194:40951), 1fec8835051845efbf4798a25ce86527 (127.27.35.195:41659)
I20250903 02:10:28.357748 30266 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49f005132e9e40809a663f747b506d19" candidate_uuid: "3f887198f79344209034c9ddf12fe026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fec8835051845efbf4798a25ce86527"
I20250903 02:10:28.357666 30192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49f005132e9e40809a663f747b506d19" candidate_uuid: "3f887198f79344209034c9ddf12fe026" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "591ccbc74880444a9b3bbf124e84e6d3"
I20250903 02:10:28.358388 30266 raft_consensus.cc:3058] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:28.358533 30192 raft_consensus.cc:3058] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:28.363461 30266 raft_consensus.cc:2466] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f887198f79344209034c9ddf12fe026 in term 1.
I20250903 02:10:28.363461 30192 raft_consensus.cc:2466] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f887198f79344209034c9ddf12fe026 in term 1.
I20250903 02:10:28.364797 30083 leader_election.cc:304] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fec8835051845efbf4798a25ce86527, 3f887198f79344209034c9ddf12fe026; no voters:
I20250903 02:10:28.365573 30314 raft_consensus.cc:2802] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:28.366508 30314 raft_consensus.cc:695] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 1 LEADER]: Becoming Leader. State: Replica: 3f887198f79344209034c9ddf12fe026, State: Running, Role: LEADER
I20250903 02:10:28.367388 30314 consensus_queue.cc:237] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.374754 30014 catalog_manager.cc:5582] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f887198f79344209034c9ddf12fe026 (127.27.35.193). New cstate: current_term: 1 leader_uuid: "3f887198f79344209034c9ddf12fe026" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } health_report { overall_health: UNKNOWN } } }
I20250903 02:10:28.492255 30182 tablet_service.cc:1515] Processing DeleteTablet for tablet 49f005132e9e40809a663f747b506d19 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:37248
I20250903 02:10:28.493780 30321 tablet_replica.cc:331] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: stopping tablet replica
I20250903 02:10:28.494411 30321 raft_consensus.cc:2241] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:28.494848 30321 raft_consensus.cc:2270] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:28.497005 30321 ts_tablet_manager.cc:1905] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250903 02:10:28.508452 30321 ts_tablet_manager.cc:1918] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 0.0
I20250903 02:10:28.508847 30321 log.cc:1199] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Deleting WAL directory at /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-1-root/wals/49f005132e9e40809a663f747b506d19
I20250903 02:10:28.513088 30322 ts_tablet_manager.cc:927] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Initiating tablet copy from peer 3f887198f79344209034c9ddf12fe026 (127.27.35.193:35771)
I20250903 02:10:28.514591 30322 tablet_copy_client.cc:323] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: tablet copy: Beginning tablet copy session from remote peer at address 127.27.35.193:35771
I20250903 02:10:28.523484 30127 tablet_copy_service.cc:140] P 3f887198f79344209034c9ddf12fe026: Received BeginTabletCopySession request for tablet 49f005132e9e40809a663f747b506d19 from peer 591ccbc74880444a9b3bbf124e84e6d3 ({username='slave'} at 127.0.0.1:38480)
I20250903 02:10:28.524049 30127 tablet_copy_service.cc:161] P 3f887198f79344209034c9ddf12fe026: Beginning new tablet copy session on tablet 49f005132e9e40809a663f747b506d19 from peer 591ccbc74880444a9b3bbf124e84e6d3 at {username='slave'} at 127.0.0.1:38480: session id = 591ccbc74880444a9b3bbf124e84e6d3-49f005132e9e40809a663f747b506d19
I20250903 02:10:28.529564 30127 tablet_copy_source_session.cc:215] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: Tablet Copy: opened 0 blocks and 1 log segments
I20250903 02:10:28.532231 30322 ts_tablet_manager.cc:1905] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250903 02:10:28.542423 30322 ts_tablet_manager.cc:1918] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 0.0
I20250903 02:10:28.543241 30322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49f005132e9e40809a663f747b506d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:28.548481 30322 tablet_copy_client.cc:806] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: tablet copy: Starting download of 0 data blocks...
I20250903 02:10:28.548966 30322 tablet_copy_client.cc:670] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: tablet copy: Starting download of 1 WAL segments...
I20250903 02:10:28.550532 30014 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60754:
table { table_id: "ba3173b9fc3947269a9874ea261ae95f" table_name: "test-table" }
I20250903 02:10:28.551116 30014 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60754:
table { table_id: "ba3173b9fc3947269a9874ea261ae95f" table_name: "test-table" }
I20250903 02:10:28.553020 30322 tablet_copy_client.cc:538] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250903 02:10:28.560102 30322 tablet_bootstrap.cc:492] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Bootstrap starting.
I20250903 02:10:28.561439 30014 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398: Sending DeleteTablet for 3 replicas of tablet 49f005132e9e40809a663f747b506d19
I20250903 02:10:28.563973 30182 tablet_service.cc:1515] Processing DeleteTablet for tablet 49f005132e9e40809a663f747b506d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 02:10:28 UTC) from {username='slave'} at 127.0.0.1:37556
I20250903 02:10:28.563890 30107 tablet_service.cc:1515] Processing DeleteTablet for tablet 49f005132e9e40809a663f747b506d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 02:10:28 UTC) from {username='slave'} at 127.0.0.1:42560
I20250903 02:10:28.563894 30256 tablet_service.cc:1515] Processing DeleteTablet for tablet 49f005132e9e40809a663f747b506d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 02:10:28 UTC) from {username='slave'} at 127.0.0.1:48576
W20250903 02:10:28.566263 29999 catalog_manager.cc:4908] TS 591ccbc74880444a9b3bbf124e84e6d3 (127.27.35.194:40951): delete failed for tablet 49f005132e9e40809a663f747b506d19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49f005132e9e40809a663f747b506d19 already in progress: copying tablet
I20250903 02:10:28.567096 30326 tablet_replica.cc:331] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527: stopping tablet replica
I20250903 02:10:28.568296 30326 raft_consensus.cc:2241] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:28.568769 30326 raft_consensus.cc:2270] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:28.568837 30325 tablet_replica.cc:331] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: stopping tablet replica
I20250903 02:10:28.569693 30325 raft_consensus.cc:2241] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:28.570611 30325 pending_rounds.cc:70] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: Trying to abort 1 pending ops.
I20250903 02:10:28.570742 30326 ts_tablet_manager.cc:1905] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 02:10:28.570991 30325 pending_rounds.cc:77] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196120794599297024 op_type: NO_OP noop_request { }
I20250903 02:10:28.571583 30325 raft_consensus.cc:2887] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 02:10:28.572132 30325 raft_consensus.cc:2270] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:28.574996 30325 ts_tablet_manager.cc:1905] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 02:10:28.576198 30322 tablet_bootstrap.cc:492] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250903 02:10:28.577152 30322 tablet_bootstrap.cc:492] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Bootstrap complete.
I20250903 02:10:28.577867 30322 ts_tablet_manager.cc:1397] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20250903 02:10:28.580880 30322 raft_consensus.cc:357] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.581827 30322 raft_consensus.cc:738] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 591ccbc74880444a9b3bbf124e84e6d3, State: Initialized, Role: FOLLOWER
I20250903 02:10:28.582902 30326 ts_tablet_manager.cc:1918] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250903 02:10:28.582597 30322 consensus_queue.cc:260] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "591ccbc74880444a9b3bbf124e84e6d3" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 40951 } } peers { permanent_uuid: "3f887198f79344209034c9ddf12fe026" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35771 } } peers { permanent_uuid: "1fec8835051845efbf4798a25ce86527" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 41659 } }
I20250903 02:10:28.583314 30326 log.cc:1199] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527: Deleting WAL directory at /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-2-root/wals/49f005132e9e40809a663f747b506d19
I20250903 02:10:28.584220 30326 ts_tablet_manager.cc:1939] T 49f005132e9e40809a663f747b506d19 P 1fec8835051845efbf4798a25ce86527: Deleting consensus metadata
I20250903 02:10:28.585700 30322 ts_tablet_manager.cc:1428] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
W20250903 02:10:28.586077 30322 ts_tablet_manager.cc:1450] Injecting 50ms of latency into OpenTablet
I20250903 02:10:28.587370 30002 catalog_manager.cc:4928] TS 1fec8835051845efbf4798a25ce86527 (127.27.35.195:41659): tablet 49f005132e9e40809a663f747b506d19 (table test-table [id=ba3173b9fc3947269a9874ea261ae95f]) successfully deleted
I20250903 02:10:28.591212 30325 ts_tablet_manager.cc:1918] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 02:10:28.591351 30182 tablet_service.cc:1515] Processing DeleteTablet for tablet 49f005132e9e40809a663f747b506d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 02:10:28 UTC) from {username='slave'} at 127.0.0.1:37556
I20250903 02:10:28.591687 30325 log.cc:1199] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: Deleting WAL directory at /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-0-root/wals/49f005132e9e40809a663f747b506d19
I20250903 02:10:28.592363 30325 ts_tablet_manager.cc:1939] T 49f005132e9e40809a663f747b506d19 P 3f887198f79344209034c9ddf12fe026: Deleting consensus metadata
W20250903 02:10:28.592713 29999 catalog_manager.cc:4908] TS 591ccbc74880444a9b3bbf124e84e6d3 (127.27.35.194:40951): delete failed for tablet 49f005132e9e40809a663f747b506d19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49f005132e9e40809a663f747b506d19 already in progress: copying tablet
I20250903 02:10:28.593559 30002 catalog_manager.cc:4928] TS 3f887198f79344209034c9ddf12fe026 (127.27.35.193:35771): tablet 49f005132e9e40809a663f747b506d19 (table test-table [id=ba3173b9fc3947269a9874ea261ae95f]) successfully deleted
I20250903 02:10:28.638343 30127 tablet_copy_service.cc:342] P 3f887198f79344209034c9ddf12fe026: Request end of tablet copy session 591ccbc74880444a9b3bbf124e84e6d3-49f005132e9e40809a663f747b506d19 received from {username='slave'} at 127.0.0.1:38480
I20250903 02:10:28.638799 30127 tablet_copy_service.cc:434] P 3f887198f79344209034c9ddf12fe026: ending tablet copy session 591ccbc74880444a9b3bbf124e84e6d3-49f005132e9e40809a663f747b506d19 on tablet 49f005132e9e40809a663f747b506d19 with peer 591ccbc74880444a9b3bbf124e84e6d3
I20250903 02:10:28.642679 30182 tablet_service.cc:1515] Processing DeleteTablet for tablet 49f005132e9e40809a663f747b506d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-03 02:10:28 UTC) from {username='slave'} at 127.0.0.1:37556
I20250903 02:10:28.643379 30321 tablet_replica.cc:331] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: stopping tablet replica
I20250903 02:10:28.643980 30321 raft_consensus.cc:2241] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:28.644337 30321 pending_rounds.cc:70] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Trying to abort 1 pending ops.
I20250903 02:10:28.644529 30321 pending_rounds.cc:77] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196120794599297024 op_type: NO_OP noop_request { }
I20250903 02:10:28.644800 30321 raft_consensus.cc:2887] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250903 02:10:28.645080 30321 raft_consensus.cc:2270] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:28.647163 30321 ts_tablet_manager.cc:1905] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250903 02:10:28.657383 30321 ts_tablet_manager.cc:1918] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250903 02:10:28.657677 30321 log.cc:1199] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Deleting WAL directory at /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TsTabletManagerITest.TestDeleteTableDuringTabletCopy.1756865376507568-27791-0/minicluster-data/ts-1-root/wals/49f005132e9e40809a663f747b506d19
I20250903 02:10:28.658280 30321 ts_tablet_manager.cc:1939] T 49f005132e9e40809a663f747b506d19 P 591ccbc74880444a9b3bbf124e84e6d3: Deleting consensus metadata
I20250903 02:10:28.660516 29999 catalog_manager.cc:4928] TS 591ccbc74880444a9b3bbf124e84e6d3 (127.27.35.194:40951): tablet 49f005132e9e40809a663f747b506d19 (table test-table [id=ba3173b9fc3947269a9874ea261ae95f]) successfully deleted
I20250903 02:10:28.699143 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:10:28.722568 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:28.729025 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:10:28.737847 27791 tablet_server.cc:178] TabletServer@127.27.35.194:0 shutting down...
I20250903 02:10:28.755798 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:28.772372 27791 tablet_server.cc:195] TabletServer@127.27.35.194:0 shutdown complete.
I20250903 02:10:28.781149 27791 tablet_server.cc:178] TabletServer@127.27.35.195:0 shutting down...
I20250903 02:10:28.797958 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:28.814396 27791 tablet_server.cc:195] TabletServer@127.27.35.195:0 shutdown complete.
I20250903 02:10:28.823082 27791 master.cc:561] Master@127.27.35.254:43777 shutting down...
I20250903 02:10:28.838001 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:28.838564 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:28.838886 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6d380471611f47469d70ee751f7d7398: stopping tablet replica
I20250903 02:10:28.847726 27791 master.cc:583] Master@127.27.35.254:43777 shutdown complete.
[ OK ] TsTabletManagerITest.TestDeleteTableDuringTabletCopy (1450 ms)
[----------] 4 tests from TsTabletManagerITest (41990 ms total)
[----------] 4 tests from TxnStatusTabletManagementTest
[ RUN ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet
I20250903 02:10:28.872903 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:40111
I20250903 02:10:28.873978 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:28.878738 30333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:28.880112 30334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:28.881314 30336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:28.881629 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:28.882647 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:28.882824 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:28.882941 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865428882929 us; error 0 us; skew 500 ppm
I20250903 02:10:28.883425 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:28.885658 27791 webserver.cc:480] Webserver started at http://127.27.35.254:46799/ using document root <none> and password file <none>
I20250903 02:10:28.886099 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:28.886250 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:28.886457 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:28.887440 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "b6b26d87a724464ca8ca69b4c09f5893"
format_stamp: "Formatted at 2025-09-03 02:10:28 on dist-test-slave-h78g"
I20250903 02:10:28.891683 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250903 02:10:28.894901 30341 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:28.895648 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:10:28.895931 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "b6b26d87a724464ca8ca69b4c09f5893"
format_stamp: "Formatted at 2025-09-03 02:10:28 on dist-test-slave-h78g"
I20250903 02:10:28.896173 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:28.923377 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:28.924386 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:28.959986 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:40111
I20250903 02:10:28.960075 30392 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:40111 every 8 connection(s)
I20250903 02:10:28.963548 30393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:28.973722 30393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893: Bootstrap starting.
I20250903 02:10:28.977941 30393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:28.981928 30393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893: No bootstrap required, opened a new log
I20250903 02:10:28.984114 30393 raft_consensus.cc:357] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b26d87a724464ca8ca69b4c09f5893" member_type: VOTER }
I20250903 02:10:28.984499 30393 raft_consensus.cc:383] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:28.984740 30393 raft_consensus.cc:738] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b26d87a724464ca8ca69b4c09f5893, State: Initialized, Role: FOLLOWER
I20250903 02:10:28.985306 30393 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b26d87a724464ca8ca69b4c09f5893" member_type: VOTER }
I20250903 02:10:28.985749 30393 raft_consensus.cc:397] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:28.985960 30393 raft_consensus.cc:491] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:28.986227 30393 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:28.990701 30393 raft_consensus.cc:513] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b26d87a724464ca8ca69b4c09f5893" member_type: VOTER }
I20250903 02:10:28.991225 30393 leader_election.cc:304] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6b26d87a724464ca8ca69b4c09f5893; no voters:
I20250903 02:10:28.992462 30393 leader_election.cc:290] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:28.992785 30396 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:28.994138 30396 raft_consensus.cc:695] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 1 LEADER]: Becoming Leader. State: Replica: b6b26d87a724464ca8ca69b4c09f5893, State: Running, Role: LEADER
I20250903 02:10:28.994825 30396 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b26d87a724464ca8ca69b4c09f5893" member_type: VOTER }
I20250903 02:10:28.995364 30393 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:10:28.997769 30397 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6b26d87a724464ca8ca69b4c09f5893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b26d87a724464ca8ca69b4c09f5893" member_type: VOTER } }
I20250903 02:10:28.997777 30398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6b26d87a724464ca8ca69b4c09f5893. Latest consensus state: current_term: 1 leader_uuid: "b6b26d87a724464ca8ca69b4c09f5893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b26d87a724464ca8ca69b4c09f5893" member_type: VOTER } }
I20250903 02:10:28.998461 30397 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:28.998658 30398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:29.004338 30401 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:10:29.011988 30401 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:10:29.013057 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:10:29.021107 30401 catalog_manager.cc:1349] Generated new cluster ID: b786fbcbce6f41e088adba23902e070e
I20250903 02:10:29.021469 30401 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:10:29.040655 30401 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:10:29.041930 30401 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:10:29.053969 30401 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893: Generated new TSK 0
I20250903 02:10:29.054610 30401 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:10:29.080293 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:29.085875 30414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:29.087173 30415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:29.088923 30417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:29.090206 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:29.090963 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:29.091140 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:29.091300 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865429091281 us; error 0 us; skew 500 ppm
I20250903 02:10:29.091861 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:29.094252 27791 webserver.cc:480] Webserver started at http://127.27.35.193:38699/ using document root <none> and password file <none>
I20250903 02:10:29.094785 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:29.094978 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:29.095235 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:29.096378 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "ed34df99d19442c6a7c6e71c6118fff9"
format_stamp: "Formatted at 2025-09-03 02:10:29 on dist-test-slave-h78g"
I20250903 02:10:29.100981 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:10:29.105211 30422 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:29.106006 27791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250903 02:10:29.106299 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "ed34df99d19442c6a7c6e71c6118fff9"
format_stamp: "Formatted at 2025-09-03 02:10:29 on dist-test-slave-h78g"
I20250903 02:10:29.106556 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:29.142371 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:29.143471 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:29.148519 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:29.149036 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.001s
I20250903 02:10:29.149473 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:29.149794 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:29.203578 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:35255
I20250903 02:10:29.203692 30492 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:35255 every 8 connection(s)
I20250903 02:10:29.208056 30493 heartbeater.cc:344] Connected to a master server at 127.27.35.254:40111
I20250903 02:10:29.208449 30493 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:29.209161 30493 heartbeater.cc:507] Master 127.27.35.254:40111 requested a full tablet report, sending...
I20250903 02:10:29.210999 30358 ts_manager.cc:194] Registered new tserver with Master: ed34df99d19442c6a7c6e71c6118fff9 (127.27.35.193:35255)
I20250903 02:10:29.211587 27791 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004929252s
I20250903 02:10:29.213294 30358 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39398
I20250903 02:10:29.245208 30458 tablet_service.cc:1468] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20250903 02:10:29.246464 30458 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:29.258746 30503 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: Bootstrap starting.
I20250903 02:10:29.263301 30503 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:29.267455 30503 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: No bootstrap required, opened a new log
I20250903 02:10:29.267916 30503 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.002s
I20250903 02:10:29.269838 30503 raft_consensus.cc:357] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } }
I20250903 02:10:29.270241 30503 raft_consensus.cc:383] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:29.270476 30503 raft_consensus.cc:738] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed34df99d19442c6a7c6e71c6118fff9, State: Initialized, Role: FOLLOWER
I20250903 02:10:29.271039 30503 consensus_queue.cc:260] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } }
I20250903 02:10:29.271487 30503 raft_consensus.cc:397] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:29.271720 30503 raft_consensus.cc:491] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:29.272017 30503 raft_consensus.cc:3058] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:29.276576 30503 raft_consensus.cc:513] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } }
I20250903 02:10:29.277153 30503 leader_election.cc:304] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed34df99d19442c6a7c6e71c6118fff9; no voters:
I20250903 02:10:29.278345 30503 leader_election.cc:290] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:29.278633 30505 raft_consensus.cc:2802] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:29.280119 30505 raft_consensus.cc:695] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 1 LEADER]: Becoming Leader. State: Replica: ed34df99d19442c6a7c6e71c6118fff9, State: Running, Role: LEADER
I20250903 02:10:29.280398 30493 heartbeater.cc:499] Master 127.27.35.254:40111 was elected leader, sending a full tablet report...
I20250903 02:10:29.281133 30503 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20250903 02:10:29.280977 30505 consensus_queue.cc:237] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } }
I20250903 02:10:29.284862 30506 tablet_replica.cc:440] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed34df99d19442c6a7c6e71c6118fff9" committed_config { opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } } }
I20250903 02:10:29.285490 30506 tablet_replica.cc:443] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: This TxnStatusTablet replica's current role is: LEADER
I20250903 02:10:29.287974 30507 tablet_replica.cc:440] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: TxnStatusTablet state changed. Reason: New leader ed34df99d19442c6a7c6e71c6118fff9. Latest consensus state: current_term: 1 leader_uuid: "ed34df99d19442c6a7c6e71c6118fff9" committed_config { opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } } }
I20250903 02:10:29.288431 30507 tablet_replica.cc:443] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: This TxnStatusTablet replica's current role is: LEADER
W20250903 02:10:29.289398 30358 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:29.290457 30510 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 02:10:29.290966 30510 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250903 02:10:29.294234 30358 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:29.325338 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:10:29.345574 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:29.346393 27791 tablet_replica.cc:331] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: stopping tablet replica
I20250903 02:10:29.347079 27791 raft_consensus.cc:2241] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:29.347666 27791 raft_consensus.cc:2270] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:29.361121 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:10:29.372395 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:29.378131 30513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:29.379244 30514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:29.382006 27791 server_base.cc:1047] running on GCE node
W20250903 02:10:29.382761 30516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:29.383921 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:29.384194 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:29.384400 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865429384382 us; error 0 us; skew 500 ppm
I20250903 02:10:29.385119 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:29.387965 27791 webserver.cc:480] Webserver started at http://127.27.35.193:38699/ using document root <none> and password file <none>
I20250903 02:10:29.388607 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:29.388841 27791 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250903 02:10:29.393422 27791 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250903 02:10:29.396942 30521 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:29.397800 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250903 02:10:29.398149 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "ed34df99d19442c6a7c6e71c6118fff9"
format_stamp: "Formatted at 2025-09-03 02:10:29 on dist-test-slave-h78g"
I20250903 02:10:29.398514 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:29.414588 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:29.415992 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:29.425382 30530 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250903 02:10:29.433499 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250903 02:10:29.433852 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250903 02:10:29.434228 27791 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250903 02:10:29.442087 27791 ts_tablet_manager.cc:610] Registered 1 tablets
I20250903 02:10:29.442428 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.008s sys 0.000s
I20250903 02:10:29.442533 30530 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: Bootstrap starting.
I20250903 02:10:29.486112 30530 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 02:10:29.487316 30530 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: Bootstrap complete.
I20250903 02:10:29.488123 30530 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: Time spent bootstrapping tablet: real 0.046s user 0.024s sys 0.018s
I20250903 02:10:29.490859 30530 raft_consensus.cc:357] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } }
I20250903 02:10:29.491515 30530 raft_consensus.cc:738] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed34df99d19442c6a7c6e71c6118fff9, State: Initialized, Role: FOLLOWER
I20250903 02:10:29.492295 30530 consensus_queue.cc:260] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } }
I20250903 02:10:29.493000 30530 raft_consensus.cc:397] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:29.493361 30530 raft_consensus.cc:491] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:29.493765 30530 raft_consensus.cc:3058] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:10:29.501390 30530 raft_consensus.cc:513] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } }
I20250903 02:10:29.502257 30530 leader_election.cc:304] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed34df99d19442c6a7c6e71c6118fff9; no voters:
I20250903 02:10:29.504067 30530 leader_election.cc:290] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250903 02:10:29.504433 30575 raft_consensus.cc:2802] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 2 FOLLOWER]: Leader election won for term 2
I20250903 02:10:29.509850 30575 raft_consensus.cc:695] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 2 LEADER]: Becoming Leader. State: Replica: ed34df99d19442c6a7c6e71c6118fff9, State: Running, Role: LEADER
I20250903 02:10:29.510807 30575 consensus_queue.cc:237] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } }
I20250903 02:10:29.515730 30530 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9: Time spent starting tablet: real 0.027s user 0.017s sys 0.008s
I20250903 02:10:29.522734 30578 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ed34df99d19442c6a7c6e71c6118fff9" committed_config { opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } } }
I20250903 02:10:29.524173 30578 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250903 02:10:29.523577 30577 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: New leader ed34df99d19442c6a7c6e71c6118fff9. Latest consensus state: current_term: 2 leader_uuid: "ed34df99d19442c6a7c6e71c6118fff9" committed_config { opid_index: -1 peers { permanent_uuid: "ed34df99d19442c6a7c6e71c6118fff9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 35255 } } }
I20250903 02:10:29.525257 30577 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250903 02:10:29.526155 30585 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 02:10:29.526612 30585 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250903 02:10:29.539647 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:35255
I20250903 02:10:29.539831 30599 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:35255 every 8 connection(s)
I20250903 02:10:29.545495 30600 heartbeater.cc:344] Connected to a master server at 127.27.35.254:40111
I20250903 02:10:29.545948 30600 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:29.547247 30600 heartbeater.cc:507] Master 127.27.35.254:40111 requested a full tablet report, sending...
I20250903 02:10:29.547710 27791 tablet_server.cc:178] TabletServer@127.27.35.193:35255 shutting down...
I20250903 02:10:29.551403 30358 ts_manager.cc:194] Re-registered known tserver with Master: ed34df99d19442c6a7c6e71c6118fff9 (127.27.35.193:35255)
W20250903 02:10:29.552333 30358 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:29.554436 30358 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39420
I20250903 02:10:29.558604 30600 heartbeater.cc:499] Master 127.27.35.254:40111 was elected leader, sending a full tablet report...
W20250903 02:10:29.561443 30358 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:29.582530 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:29.583264 27791 tablet_replica.cc:331] stopping tablet replica
I20250903 02:10:29.583904 27791 raft_consensus.cc:2241] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 2 LEADER]: Raft consensus shutting down.
I20250903 02:10:29.584342 27791 raft_consensus.cc:2270] T 11111111111111111111111111111111 P ed34df99d19442c6a7c6e71c6118fff9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:29.604797 27791 tablet_server.cc:195] TabletServer@127.27.35.193:35255 shutdown complete.
I20250903 02:10:29.618508 27791 master.cc:561] Master@127.27.35.254:40111 shutting down...
I20250903 02:10:29.636864 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:29.637400 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:29.637699 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P b6b26d87a724464ca8ca69b4c09f5893: stopping tablet replica
I20250903 02:10:29.655881 27791 master.cc:583] Master@127.27.35.254:40111 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestBootstrapTransactionStatusTablet (806 ms)
[ RUN ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet
I20250903 02:10:29.680723 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:36845
I20250903 02:10:29.681819 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:29.687146 30606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:29.688266 30607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:29.690186 30609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:29.691602 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:29.692648 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:29.692905 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:29.693084 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865429693069 us; error 0 us; skew 500 ppm
I20250903 02:10:29.693794 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:29.696784 27791 webserver.cc:480] Webserver started at http://127.27.35.254:45813/ using document root <none> and password file <none>
I20250903 02:10:29.697398 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:29.697628 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:29.697948 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:29.699424 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "e7e03197bca24e0f96d1c00481f7ae59"
format_stamp: "Formatted at 2025-09-03 02:10:29 on dist-test-slave-h78g"
I20250903 02:10:29.705518 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250903 02:10:29.710031 30614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:29.711076 27791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250903 02:10:29.711493 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "e7e03197bca24e0f96d1c00481f7ae59"
format_stamp: "Formatted at 2025-09-03 02:10:29 on dist-test-slave-h78g"
I20250903 02:10:29.711910 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:29.727324 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:29.728710 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:29.782769 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:36845
I20250903 02:10:29.782859 30665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:36845 every 8 connection(s)
I20250903 02:10:29.787158 30666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:29.797482 30666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59: Bootstrap starting.
I20250903 02:10:29.801937 30666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:29.806174 30666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59: No bootstrap required, opened a new log
I20250903 02:10:29.808257 30666 raft_consensus.cc:357] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e03197bca24e0f96d1c00481f7ae59" member_type: VOTER }
I20250903 02:10:29.808655 30666 raft_consensus.cc:383] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:29.808876 30666 raft_consensus.cc:738] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e03197bca24e0f96d1c00481f7ae59, State: Initialized, Role: FOLLOWER
I20250903 02:10:29.809432 30666 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e03197bca24e0f96d1c00481f7ae59" member_type: VOTER }
I20250903 02:10:29.809880 30666 raft_consensus.cc:397] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:29.810093 30666 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:29.810340 30666 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:29.815014 30666 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e03197bca24e0f96d1c00481f7ae59" member_type: VOTER }
I20250903 02:10:29.815711 30666 leader_election.cc:304] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7e03197bca24e0f96d1c00481f7ae59; no voters:
I20250903 02:10:29.817142 30666 leader_election.cc:290] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:29.817495 30669 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:29.819370 30669 raft_consensus.cc:695] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 1 LEADER]: Becoming Leader. State: Replica: e7e03197bca24e0f96d1c00481f7ae59, State: Running, Role: LEADER
I20250903 02:10:29.820242 30669 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e03197bca24e0f96d1c00481f7ae59" member_type: VOTER }
I20250903 02:10:29.821033 30666 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:10:29.823472 30670 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7e03197bca24e0f96d1c00481f7ae59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e03197bca24e0f96d1c00481f7ae59" member_type: VOTER } }
I20250903 02:10:29.823560 30671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7e03197bca24e0f96d1c00481f7ae59. Latest consensus state: current_term: 1 leader_uuid: "e7e03197bca24e0f96d1c00481f7ae59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e03197bca24e0f96d1c00481f7ae59" member_type: VOTER } }
I20250903 02:10:29.824204 30670 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:29.824285 30671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:29.828573 30674 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:10:29.838346 30674 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:10:29.848421 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:10:29.851646 30674 catalog_manager.cc:1349] Generated new cluster ID: 115589755d0746c7b00fd15679e45582
I20250903 02:10:29.851997 30674 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:10:29.873768 30674 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:10:29.875123 30674 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:10:29.891631 30674 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59: Generated new TSK 0
I20250903 02:10:29.892342 30674 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:10:29.915159 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:29.922055 30687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:29.923230 30688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:29.924715 30690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:29.926064 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:29.927167 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:29.927466 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:29.927670 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865429927652 us; error 0 us; skew 500 ppm
I20250903 02:10:29.928411 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:29.931305 27791 webserver.cc:480] Webserver started at http://127.27.35.193:45371/ using document root <none> and password file <none>
I20250903 02:10:29.931989 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:29.932231 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:29.932564 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:29.934036 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "02f0a7bb98574ea78e6b3e27e2441914"
format_stamp: "Formatted at 2025-09-03 02:10:29 on dist-test-slave-h78g"
I20250903 02:10:29.940270 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250903 02:10:29.944775 30695 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:29.945658 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20250903 02:10:29.945950 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "02f0a7bb98574ea78e6b3e27e2441914"
format_stamp: "Formatted at 2025-09-03 02:10:29 on dist-test-slave-h78g"
I20250903 02:10:29.946236 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:29.955961 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:29.956948 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:29.977080 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:29.977361 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:29.977608 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:29.977769 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:30.048682 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:44769
I20250903 02:10:30.048776 30765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:44769 every 8 connection(s)
I20250903 02:10:30.053457 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250903 02:10:30.053694 30766 heartbeater.cc:344] Connected to a master server at 127.27.35.254:36845
I20250903 02:10:30.054049 30766 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:30.054890 30766 heartbeater.cc:507] Master 127.27.35.254:36845 requested a full tablet report, sending...
I20250903 02:10:30.057049 30631 ts_manager.cc:194] Registered new tserver with Master: 02f0a7bb98574ea78e6b3e27e2441914 (127.27.35.193:44769)
I20250903 02:10:30.059369 30631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52082
W20250903 02:10:30.060364 30768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:30.063196 30769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:30.065928 30771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:30.066727 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:30.067422 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:30.067618 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:30.067793 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865430067776 us; error 0 us; skew 500 ppm
I20250903 02:10:30.068269 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:30.071079 27791 webserver.cc:480] Webserver started at http://127.27.35.194:40203/ using document root <none> and password file <none>
I20250903 02:10:30.071538 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:30.071710 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:30.071992 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:30.073014 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-1-root/instance:
uuid: "68ea3861082f4791925de9b1ebe01b2c"
format_stamp: "Formatted at 2025-09-03 02:10:30 on dist-test-slave-h78g"
I20250903 02:10:30.077471 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:10:30.082059 30776 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:30.083315 27791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250903 02:10:30.083699 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-1-root
uuid: "68ea3861082f4791925de9b1ebe01b2c"
format_stamp: "Formatted at 2025-09-03 02:10:30 on dist-test-slave-h78g"
I20250903 02:10:30.084133 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:30.143401 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:30.144821 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:30.151940 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:30.152305 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:30.152611 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:30.152854 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250903 02:10:30.218513 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.194:40385
I20250903 02:10:30.218660 30845 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.194:40385 every 8 connection(s)
I20250903 02:10:30.224874 30846 heartbeater.cc:344] Connected to a master server at 127.27.35.254:36845
I20250903 02:10:30.225337 30846 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:30.226250 30846 heartbeater.cc:507] Master 127.27.35.254:36845 requested a full tablet report, sending...
I20250903 02:10:30.228667 30631 ts_manager.cc:194] Registered new tserver with Master: 68ea3861082f4791925de9b1ebe01b2c (127.27.35.194:40385)
I20250903 02:10:30.229216 27791 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.006252082s
I20250903 02:10:30.230511 30631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52112
I20250903 02:10:30.263449 30731 tablet_service.cc:1468] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20250903 02:10:30.264655 30731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:30.276058 30856 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: Bootstrap starting.
I20250903 02:10:30.280567 30856 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:30.284714 30856 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: No bootstrap required, opened a new log
I20250903 02:10:30.285123 30856 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20250903 02:10:30.287204 30856 raft_consensus.cc:357] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } }
I20250903 02:10:30.287631 30856 raft_consensus.cc:383] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:30.287894 30856 raft_consensus.cc:738] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f0a7bb98574ea78e6b3e27e2441914, State: Initialized, Role: FOLLOWER
I20250903 02:10:30.288403 30856 consensus_queue.cc:260] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } }
I20250903 02:10:30.288861 30856 raft_consensus.cc:397] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:30.289098 30856 raft_consensus.cc:491] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:30.289337 30856 raft_consensus.cc:3058] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:30.293743 30856 raft_consensus.cc:513] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } }
I20250903 02:10:30.294294 30856 leader_election.cc:304] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02f0a7bb98574ea78e6b3e27e2441914; no voters:
I20250903 02:10:30.295691 30856 leader_election.cc:290] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:30.296065 30858 raft_consensus.cc:2802] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:30.297688 30766 heartbeater.cc:499] Master 127.27.35.254:36845 was elected leader, sending a full tablet report...
I20250903 02:10:30.298741 30856 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20250903 02:10:30.298946 30858 raft_consensus.cc:695] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 1 LEADER]: Becoming Leader. State: Replica: 02f0a7bb98574ea78e6b3e27e2441914, State: Running, Role: LEADER
I20250903 02:10:30.299728 30858 consensus_queue.cc:237] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } }
I20250903 02:10:30.303042 30859 tablet_replica.cc:440] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02f0a7bb98574ea78e6b3e27e2441914" committed_config { opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } } }
I20250903 02:10:30.303655 30859 tablet_replica.cc:443] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: This TxnStatusTablet replica's current role is: LEADER
W20250903 02:10:30.307709 30631 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:30.307873 30860 tablet_replica.cc:440] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: TxnStatusTablet state changed. Reason: New leader 02f0a7bb98574ea78e6b3e27e2441914. Latest consensus state: current_term: 1 leader_uuid: "02f0a7bb98574ea78e6b3e27e2441914" committed_config { opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } } }
I20250903 02:10:30.308790 30860 tablet_replica.cc:443] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: This TxnStatusTablet replica's current role is: LEADER
I20250903 02:10:30.312630 30862 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 02:10:30.313076 30862 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250903 02:10:30.313649 30631 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:30.364876 30867 ts_tablet_manager.cc:927] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: Initiating tablet copy from peer 02f0a7bb98574ea78e6b3e27e2441914 (127.27.35.193:44769)
I20250903 02:10:30.366662 30867 tablet_copy_client.cc:323] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.35.193:44769
I20250903 02:10:30.375593 30751 tablet_copy_service.cc:140] P 02f0a7bb98574ea78e6b3e27e2441914: Received BeginTabletCopySession request for tablet 11111111111111111111111111111111 from peer 68ea3861082f4791925de9b1ebe01b2c ({username='slave'} at 127.0.0.1:49552)
I20250903 02:10:30.376119 30751 tablet_copy_service.cc:161] P 02f0a7bb98574ea78e6b3e27e2441914: Beginning new tablet copy session on tablet 11111111111111111111111111111111 from peer 68ea3861082f4791925de9b1ebe01b2c at {username='slave'} at 127.0.0.1:49552: session id = 68ea3861082f4791925de9b1ebe01b2c-11111111111111111111111111111111
I20250903 02:10:30.381026 30751 tablet_copy_source_session.cc:215] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: Tablet Copy: opened 0 blocks and 1 log segments
I20250903 02:10:30.383597 30867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:30.394870 30867 tablet_copy_client.cc:806] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: tablet copy: Starting download of 0 data blocks...
I20250903 02:10:30.395380 30867 tablet_copy_client.cc:670] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: tablet copy: Starting download of 1 WAL segments...
I20250903 02:10:30.398955 30867 tablet_copy_client.cc:538] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250903 02:10:30.405515 30867 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: Bootstrap starting.
I20250903 02:10:30.428825 30867 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 02:10:30.429638 30867 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: Bootstrap complete.
I20250903 02:10:30.430191 30867 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.012s
I20250903 02:10:30.432179 30867 raft_consensus.cc:357] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } }
I20250903 02:10:30.432633 30867 raft_consensus.cc:738] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 68ea3861082f4791925de9b1ebe01b2c, State: Initialized, Role: NON_PARTICIPANT
I20250903 02:10:30.433211 30867 consensus_queue.cc:260] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } }
I20250903 02:10:30.435053 30870 tablet_replica.cc:440] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } } }
I20250903 02:10:30.435252 30846 heartbeater.cc:499] Master 127.27.35.254:36845 was elected leader, sending a full tablet report...
I20250903 02:10:30.435621 30870 tablet_replica.cc:443] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: This TxnStatusTablet replica's current role is: NON_PARTICIPANT
I20250903 02:10:30.435889 30867 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20250903 02:10:30.437758 30751 tablet_copy_service.cc:342] P 02f0a7bb98574ea78e6b3e27e2441914: Request end of tablet copy session 68ea3861082f4791925de9b1ebe01b2c-11111111111111111111111111111111 received from {username='slave'} at 127.0.0.1:49552
I20250903 02:10:30.438195 30751 tablet_copy_service.cc:434] P 02f0a7bb98574ea78e6b3e27e2441914: ending tablet copy session 68ea3861082f4791925de9b1ebe01b2c-11111111111111111111111111111111 on tablet 11111111111111111111111111111111 with peer 68ea3861082f4791925de9b1ebe01b2c
W20250903 02:10:30.438409 30631 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
W20250903 02:10:30.442474 30631 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:30.460117 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:10:30.485412 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:30.486289 27791 tablet_replica.cc:331] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: stopping tablet replica
I20250903 02:10:30.487087 27791 raft_consensus.cc:2241] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:30.487686 27791 raft_consensus.cc:2270] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:30.509970 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:10:30.521530 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:30.527947 30871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:30.529249 30872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:30.531296 27791 server_base.cc:1047] running on GCE node
W20250903 02:10:30.532171 30875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:30.533164 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:30.533416 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:30.533586 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865430533570 us; error 0 us; skew 500 ppm
I20250903 02:10:30.534087 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:30.536948 27791 webserver.cc:480] Webserver started at http://127.27.35.193:45371/ using document root <none> and password file <none>
I20250903 02:10:30.537436 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:30.537600 27791 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250903 02:10:30.541379 27791 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250903 02:10:30.544288 30879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:30.545128 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 02:10:30.545414 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "02f0a7bb98574ea78e6b3e27e2441914"
format_stamp: "Formatted at 2025-09-03 02:10:29 on dist-test-slave-h78g"
I20250903 02:10:30.545688 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:30.568629 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:30.569623 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:30.578508 30888 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250903 02:10:30.586436 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250903 02:10:30.586783 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.001s
I20250903 02:10:30.587112 27791 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250903 02:10:30.594589 27791 ts_tablet_manager.cc:610] Registered 1 tablets
I20250903 02:10:30.594859 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.008s sys 0.000s
I20250903 02:10:30.595150 30888 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: Bootstrap starting.
I20250903 02:10:30.635129 30888 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 02:10:30.636338 30888 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: Bootstrap complete.
I20250903 02:10:30.637120 30888 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: Time spent bootstrapping tablet: real 0.042s user 0.030s sys 0.011s
I20250903 02:10:30.639775 30888 raft_consensus.cc:357] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } }
I20250903 02:10:30.640467 30888 raft_consensus.cc:738] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f0a7bb98574ea78e6b3e27e2441914, State: Initialized, Role: FOLLOWER
I20250903 02:10:30.641220 30888 consensus_queue.cc:260] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } }
I20250903 02:10:30.641989 30888 raft_consensus.cc:397] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:30.642292 30888 raft_consensus.cc:491] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:30.642663 30888 raft_consensus.cc:3058] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:10:30.650063 30888 raft_consensus.cc:513] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } }
I20250903 02:10:30.650666 30888 leader_election.cc:304] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02f0a7bb98574ea78e6b3e27e2441914; no voters:
I20250903 02:10:30.652799 30888 leader_election.cc:290] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250903 02:10:30.653100 30937 raft_consensus.cc:2802] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 2 FOLLOWER]: Leader election won for term 2
I20250903 02:10:30.656546 30937 raft_consensus.cc:695] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 2 LEADER]: Becoming Leader. State: Replica: 02f0a7bb98574ea78e6b3e27e2441914, State: Running, Role: LEADER
I20250903 02:10:30.657351 30937 consensus_queue.cc:237] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } }
I20250903 02:10:30.664005 30888 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914: Time spent starting tablet: real 0.026s user 0.016s sys 0.010s
I20250903 02:10:30.666764 30940 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: New leader 02f0a7bb98574ea78e6b3e27e2441914. Latest consensus state: current_term: 2 leader_uuid: "02f0a7bb98574ea78e6b3e27e2441914" committed_config { opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } } }
I20250903 02:10:30.667546 30940 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250903 02:10:30.667866 30941 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "02f0a7bb98574ea78e6b3e27e2441914" committed_config { opid_index: -1 peers { permanent_uuid: "02f0a7bb98574ea78e6b3e27e2441914" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 44769 } } }
I20250903 02:10:30.668394 30941 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250903 02:10:30.671191 30946 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 02:10:30.671663 30946 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250903 02:10:30.687199 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:44769
I20250903 02:10:30.687323 30957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:44769 every 8 connection(s)
I20250903 02:10:30.693652 27791 tablet_server.cc:178] TabletServer@127.27.35.193:44769 shutting down...
I20250903 02:10:30.696046 30958 heartbeater.cc:344] Connected to a master server at 127.27.35.254:36845
I20250903 02:10:30.697090 30958 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:30.698324 30958 heartbeater.cc:507] Master 127.27.35.254:36845 requested a full tablet report, sending...
I20250903 02:10:30.701465 30631 ts_manager.cc:194] Re-registered known tserver with Master: 02f0a7bb98574ea78e6b3e27e2441914 (127.27.35.193:44769)
W20250903 02:10:30.702096 30631 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:30.704037 30631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52144
I20250903 02:10:30.710566 30958 heartbeater.cc:499] Master 127.27.35.254:36845 was elected leader, sending a full tablet report...
W20250903 02:10:30.716193 30631 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:30.729835 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:30.730641 27791 tablet_replica.cc:331] stopping tablet replica
I20250903 02:10:30.731181 27791 raft_consensus.cc:2241] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 2 LEADER]: Raft consensus shutting down.
I20250903 02:10:30.731633 27791 raft_consensus.cc:2270] T 11111111111111111111111111111111 P 02f0a7bb98574ea78e6b3e27e2441914 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:30.744254 27791 tablet_server.cc:195] TabletServer@127.27.35.193:44769 shutdown complete.
I20250903 02:10:30.755362 27791 tablet_server.cc:178] TabletServer@127.27.35.194:0 shutting down...
I20250903 02:10:30.782452 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:30.783290 27791 tablet_replica.cc:331] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c: stopping tablet replica
I20250903 02:10:30.783974 27791 raft_consensus.cc:2241] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20250903 02:10:30.784482 27791 raft_consensus.cc:2270] T 11111111111111111111111111111111 P 68ea3861082f4791925de9b1ebe01b2c [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20250903 02:10:30.794263 27791 tablet_server.cc:195] TabletServer@127.27.35.194:0 shutdown complete.
I20250903 02:10:30.804914 27791 master.cc:561] Master@127.27.35.254:36845 shutting down...
I20250903 02:10:30.820958 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:30.821483 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:30.821786 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P e7e03197bca24e0f96d1c00481f7ae59: stopping tablet replica
I20250903 02:10:30.842005 27791 master.cc:583] Master@127.27.35.254:36845 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestCopyTransactionStatusTablet (1194 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls
I20250903 02:10:30.877031 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:44703
I20250903 02:10:30.878572 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:30.885107 30964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:30.886233 30965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:30.889473 27791 server_base.cc:1047] running on GCE node
W20250903 02:10:30.889551 30967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:30.890616 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:30.890825 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:30.890974 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865430890958 us; error 0 us; skew 500 ppm
I20250903 02:10:30.891476 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:30.894137 27791 webserver.cc:480] Webserver started at http://127.27.35.254:33457/ using document root <none> and password file <none>
I20250903 02:10:30.894654 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:30.894838 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:30.895084 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:30.896229 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "d621b9cd3872446aae8aa3a211d7c28d"
format_stamp: "Formatted at 2025-09-03 02:10:30 on dist-test-slave-h78g"
I20250903 02:10:30.900861 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250903 02:10:30.904435 30972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:30.905268 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 02:10:30.905552 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "d621b9cd3872446aae8aa3a211d7c28d"
format_stamp: "Formatted at 2025-09-03 02:10:30 on dist-test-slave-h78g"
I20250903 02:10:30.905825 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:30.923974 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:30.924996 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:30.970475 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:44703
I20250903 02:10:30.970599 31023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:44703 every 8 connection(s)
I20250903 02:10:30.975174 31024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:30.985450 31024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d: Bootstrap starting.
I20250903 02:10:30.990034 31024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:30.994184 31024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d: No bootstrap required, opened a new log
I20250903 02:10:30.996140 31024 raft_consensus.cc:357] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621b9cd3872446aae8aa3a211d7c28d" member_type: VOTER }
I20250903 02:10:30.996526 31024 raft_consensus.cc:383] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:30.996747 31024 raft_consensus.cc:738] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d621b9cd3872446aae8aa3a211d7c28d, State: Initialized, Role: FOLLOWER
I20250903 02:10:30.997277 31024 consensus_queue.cc:260] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621b9cd3872446aae8aa3a211d7c28d" member_type: VOTER }
I20250903 02:10:30.997694 31024 raft_consensus.cc:397] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:30.997900 31024 raft_consensus.cc:491] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:30.998147 31024 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:31.002506 31024 raft_consensus.cc:513] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621b9cd3872446aae8aa3a211d7c28d" member_type: VOTER }
I20250903 02:10:31.002998 31024 leader_election.cc:304] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d621b9cd3872446aae8aa3a211d7c28d; no voters:
I20250903 02:10:31.004184 31024 leader_election.cc:290] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:31.004550 31027 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:31.006103 31027 raft_consensus.cc:695] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 1 LEADER]: Becoming Leader. State: Replica: d621b9cd3872446aae8aa3a211d7c28d, State: Running, Role: LEADER
I20250903 02:10:31.006829 31027 consensus_queue.cc:237] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621b9cd3872446aae8aa3a211d7c28d" member_type: VOTER }
I20250903 02:10:31.007284 31024 sys_catalog.cc:564] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:10:31.010751 31028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d621b9cd3872446aae8aa3a211d7c28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621b9cd3872446aae8aa3a211d7c28d" member_type: VOTER } }
I20250903 02:10:31.010947 31029 sys_catalog.cc:455] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d621b9cd3872446aae8aa3a211d7c28d. Latest consensus state: current_term: 1 leader_uuid: "d621b9cd3872446aae8aa3a211d7c28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621b9cd3872446aae8aa3a211d7c28d" member_type: VOTER } }
I20250903 02:10:31.011538 31029 sys_catalog.cc:458] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:31.011466 31028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:31.027334 31033 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:10:31.034085 31033 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:10:31.035204 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:10:31.044543 31033 catalog_manager.cc:1349] Generated new cluster ID: 1ac09459e6914fcd8cbba2fae264b3bb
I20250903 02:10:31.044945 31033 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:10:31.074389 31033 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:10:31.076153 31033 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:10:31.095062 31033 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d: Generated new TSK 0
I20250903 02:10:31.095834 31033 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:10:31.101454 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:31.107693 31045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:31.109100 31046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:31.112546 27791 server_base.cc:1047] running on GCE node
W20250903 02:10:31.114080 31048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:31.115110 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:31.115319 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:31.115478 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865431115461 us; error 0 us; skew 500 ppm
I20250903 02:10:31.116025 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:31.118808 27791 webserver.cc:480] Webserver started at http://127.27.35.193:38527/ using document root <none> and password file <none>
I20250903 02:10:31.119279 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:31.119465 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:31.119719 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:31.120863 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "476190cd7a8545cca990b6fc263178d9"
format_stamp: "Formatted at 2025-09-03 02:10:31 on dist-test-slave-h78g"
I20250903 02:10:31.125950 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250903 02:10:31.131269 31053 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:31.132637 27791 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250903 02:10:31.133044 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "476190cd7a8545cca990b6fc263178d9"
format_stamp: "Formatted at 2025-09-03 02:10:31 on dist-test-slave-h78g"
I20250903 02:10:31.133461 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCalls.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:31.148152 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:31.149121 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:31.161545 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:31.161934 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:31.162376 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:31.162644 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:31.234077 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:38639
I20250903 02:10:31.234197 31123 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:38639 every 8 connection(s)
I20250903 02:10:31.239434 31124 heartbeater.cc:344] Connected to a master server at 127.27.35.254:44703
I20250903 02:10:31.239833 31124 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:31.240542 31124 heartbeater.cc:507] Master 127.27.35.254:44703 requested a full tablet report, sending...
I20250903 02:10:31.242594 30989 ts_manager.cc:194] Registered new tserver with Master: 476190cd7a8545cca990b6fc263178d9 (127.27.35.193:38639)
I20250903 02:10:31.242842 27791 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.005023141s
I20250903 02:10:31.245154 30989 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36484
I20250903 02:10:31.279111 31089 tablet_service.cc:1468] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20250903 02:10:31.280319 31089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:31.292768 31134 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9: Bootstrap starting.
I20250903 02:10:31.297117 31134 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:31.301211 31134 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9: No bootstrap required, opened a new log
I20250903 02:10:31.301614 31134 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20250903 02:10:31.303414 31134 raft_consensus.cc:357] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "476190cd7a8545cca990b6fc263178d9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 38639 } }
I20250903 02:10:31.303858 31134 raft_consensus.cc:383] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:31.304070 31134 raft_consensus.cc:738] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476190cd7a8545cca990b6fc263178d9, State: Initialized, Role: FOLLOWER
I20250903 02:10:31.304615 31134 consensus_queue.cc:260] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "476190cd7a8545cca990b6fc263178d9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 38639 } }
I20250903 02:10:31.305047 31134 raft_consensus.cc:397] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:31.305255 31134 raft_consensus.cc:491] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:31.305492 31134 raft_consensus.cc:3058] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:31.310169 31134 raft_consensus.cc:513] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "476190cd7a8545cca990b6fc263178d9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 38639 } }
I20250903 02:10:31.310719 31134 leader_election.cc:304] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 476190cd7a8545cca990b6fc263178d9; no voters:
I20250903 02:10:31.312098 31134 leader_election.cc:290] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:31.312453 31136 raft_consensus.cc:2802] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:31.314014 31124 heartbeater.cc:499] Master 127.27.35.254:44703 was elected leader, sending a full tablet report...
I20250903 02:10:31.315109 31134 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20250903 02:10:31.315250 31136 raft_consensus.cc:695] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 1 LEADER]: Becoming Leader. State: Replica: 476190cd7a8545cca990b6fc263178d9, State: Running, Role: LEADER
I20250903 02:10:31.316066 31136 consensus_queue.cc:237] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "476190cd7a8545cca990b6fc263178d9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 38639 } }
I20250903 02:10:31.319378 31137 tablet_replica.cc:440] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "476190cd7a8545cca990b6fc263178d9" committed_config { opid_index: -1 peers { permanent_uuid: "476190cd7a8545cca990b6fc263178d9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 38639 } } }
I20250903 02:10:31.319984 31137 tablet_replica.cc:443] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9: This TxnStatusTablet replica's current role is: LEADER
I20250903 02:10:31.321373 31138 tablet_replica.cc:440] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9: TxnStatusTablet state changed. Reason: New leader 476190cd7a8545cca990b6fc263178d9. Latest consensus state: current_term: 1 leader_uuid: "476190cd7a8545cca990b6fc263178d9" committed_config { opid_index: -1 peers { permanent_uuid: "476190cd7a8545cca990b6fc263178d9" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 38639 } } }
I20250903 02:10:31.321705 31140 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
W20250903 02:10:31.321915 30989 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:31.322844 31138 tablet_replica.cc:443] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9: This TxnStatusTablet replica's current role is: LEADER
I20250903 02:10:31.327517 31140 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250903 02:10:31.328025 30989 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
W20250903 02:10:31.376528 31079 meta_cache.cc:788] Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20250903 02:10:31.376886 31079 txn_status_manager.cc:244] Participant participant of txn 1 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20250903 02:10:31.377249 31079 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20250903 02:10:31.385136 31079 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20250903 02:10:31.437310 31079 txn_status_manager.cc:244] Participant participant of txn 2 returned error for BEGIN_COMMIT op, aborting: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
I20250903 02:10:31.437692 31079 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 2
I20250903 02:10:31.447604 31079 txn_status_manager.cc:337] Participant participant was not found for ABORT_TXN, proceeding as if op succeeded: Not found: LookupRpcById { tablet: 'participant', attempt: 1 } failed
W20250903 02:10:31.464716 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39704
W20250903 02:10:31.467429 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39704
W20250903 02:10:31.469931 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39704
W20250903 02:10:31.472357 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39704
W20250903 02:10:31.475121 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39704
W20250903 02:10:31.477624 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='hughie'} at 127.0.0.1:39704
W20250903 02:10:31.487577 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39712
W20250903 02:10:31.489737 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39712
W20250903 02:10:31.491581 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39712
W20250903 02:10:31.493461 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39712
W20250903 02:10:31.495199 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39712
W20250903 02:10:31.497040 31089 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerAdminService.CoordinateTransaction from {username='butcher'} at 127.0.0.1:39712
I20250903 02:10:31.500207 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:10:31.537341 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:31.538611 27791 tablet_replica.cc:331] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9: stopping tablet replica
I20250903 02:10:31.539295 27791 raft_consensus.cc:2241] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:31.539963 27791 raft_consensus.cc:2270] T 11111111111111111111111111111111 P 476190cd7a8545cca990b6fc263178d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:31.553936 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:10:31.566896 27791 master.cc:561] Master@127.27.35.254:44703 shutting down...
I20250903 02:10:31.583968 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:31.584573 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:31.585047 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P d621b9cd3872446aae8aa3a211d7c28d: stopping tablet replica
I20250903 02:10:31.604148 27791 master.cc:583] Master@127.27.35.254:44703 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCalls (756 ms)
[ RUN ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors
I20250903 02:10:31.632325 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:45237
I20250903 02:10:31.633497 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:31.639070 31149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:31.639577 31150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:31.642764 27791 server_base.cc:1047] running on GCE node
W20250903 02:10:31.642818 31152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:31.644095 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:31.644343 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:31.644505 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865431644481 us; error 0 us; skew 500 ppm
I20250903 02:10:31.645071 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:31.647979 27791 webserver.cc:480] Webserver started at http://127.27.35.254:38875/ using document root <none> and password file <none>
I20250903 02:10:31.648470 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:31.648667 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:31.648928 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:31.650028 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "a1400cb822e94ff5805e09b4db7e62d3"
format_stamp: "Formatted at 2025-09-03 02:10:31 on dist-test-slave-h78g"
I20250903 02:10:31.655016 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20250903 02:10:31.659072 31157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:31.660375 27791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250903 02:10:31.660951 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "a1400cb822e94ff5805e09b4db7e62d3"
format_stamp: "Formatted at 2025-09-03 02:10:31 on dist-test-slave-h78g"
I20250903 02:10:31.661513 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:31.689054 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:31.690316 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:31.733868 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:45237
I20250903 02:10:31.733969 31208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:45237 every 8 connection(s)
I20250903 02:10:31.737787 31209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:31.748848 31209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3: Bootstrap starting.
I20250903 02:10:31.753400 31209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:31.757411 31209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3: No bootstrap required, opened a new log
I20250903 02:10:31.759292 31209 raft_consensus.cc:357] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1400cb822e94ff5805e09b4db7e62d3" member_type: VOTER }
I20250903 02:10:31.759672 31209 raft_consensus.cc:383] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:31.759923 31209 raft_consensus.cc:738] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1400cb822e94ff5805e09b4db7e62d3, State: Initialized, Role: FOLLOWER
I20250903 02:10:31.760452 31209 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1400cb822e94ff5805e09b4db7e62d3" member_type: VOTER }
I20250903 02:10:31.760885 31209 raft_consensus.cc:397] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:31.761073 31209 raft_consensus.cc:491] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:31.761286 31209 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:31.766018 31209 raft_consensus.cc:513] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1400cb822e94ff5805e09b4db7e62d3" member_type: VOTER }
I20250903 02:10:31.766520 31209 leader_election.cc:304] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1400cb822e94ff5805e09b4db7e62d3; no voters:
I20250903 02:10:31.767603 31209 leader_election.cc:290] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:31.767961 31212 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:31.769340 31212 raft_consensus.cc:695] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 1 LEADER]: Becoming Leader. State: Replica: a1400cb822e94ff5805e09b4db7e62d3, State: Running, Role: LEADER
I20250903 02:10:31.770037 31212 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1400cb822e94ff5805e09b4db7e62d3" member_type: VOTER }
I20250903 02:10:31.770680 31209 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:10:31.773211 31213 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1400cb822e94ff5805e09b4db7e62d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1400cb822e94ff5805e09b4db7e62d3" member_type: VOTER } }
I20250903 02:10:31.773311 31214 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1400cb822e94ff5805e09b4db7e62d3. Latest consensus state: current_term: 1 leader_uuid: "a1400cb822e94ff5805e09b4db7e62d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1400cb822e94ff5805e09b4db7e62d3" member_type: VOTER } }
I20250903 02:10:31.774061 31214 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:31.774092 31213 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:31.776687 31219 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:10:31.782815 31219 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:10:31.784204 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:10:31.791711 31219 catalog_manager.cc:1349] Generated new cluster ID: e16f68f7c70e449a8959a74744073a21
I20250903 02:10:31.792016 31219 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:10:31.806331 31219 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:10:31.807567 31219 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:10:31.827651 31219 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3: Generated new TSK 0
I20250903 02:10:31.828246 31219 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:10:31.850931 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:31.857589 31230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:31.858774 31231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:31.859323 27791 server_base.cc:1047] running on GCE node
W20250903 02:10:31.859225 31233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:31.860450 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:31.860661 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:31.860795 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865431860783 us; error 0 us; skew 500 ppm
I20250903 02:10:31.861250 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:31.863551 27791 webserver.cc:480] Webserver started at http://127.27.35.193:38433/ using document root <none> and password file <none>
I20250903 02:10:31.864106 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:31.864295 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:31.864537 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:31.865556 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "74ce44c58e3944eea01d13eea6b76e61"
format_stamp: "Formatted at 2025-09-03 02:10:31 on dist-test-slave-h78g"
I20250903 02:10:31.870179 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250903 02:10:31.873929 31238 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:31.874804 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 02:10:31.875114 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "74ce44c58e3944eea01d13eea6b76e61"
format_stamp: "Formatted at 2025-09-03 02:10:31 on dist-test-slave-h78g"
I20250903 02:10:31.875439 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:31.893011 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:31.894025 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:31.900321 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:31.900576 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:31.900871 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:31.901074 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:31.979436 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:40241
I20250903 02:10:31.979530 31308 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:40241 every 8 connection(s)
I20250903 02:10:31.984692 31309 heartbeater.cc:344] Connected to a master server at 127.27.35.254:45237
I20250903 02:10:31.985033 31309 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:31.985672 31309 heartbeater.cc:507] Master 127.27.35.254:45237 requested a full tablet report, sending...
I20250903 02:10:31.987553 31174 ts_manager.cc:194] Registered new tserver with Master: 74ce44c58e3944eea01d13eea6b76e61 (127.27.35.193:40241)
I20250903 02:10:31.988087 27791 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.005023171s
I20250903 02:10:31.990007 31174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54112
I20250903 02:10:32.020450 31274 tablet_service.cc:1468] Processing CreateTablet for tablet 11111111111111111111111111111111 (TXN_STATUS_TABLE table=11111111111111111111111111111111_table_name [id=11111111111111111111111111111111_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20250903 02:10:32.021615 31274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11111111111111111111111111111111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:32.034559 31319 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61: Bootstrap starting.
I20250903 02:10:32.042727 31319 tablet_bootstrap.cc:654] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:32.047530 31319 tablet_bootstrap.cc:492] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61: No bootstrap required, opened a new log
I20250903 02:10:32.048349 31319 ts_tablet_manager.cc:1397] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20250903 02:10:32.050200 31319 raft_consensus.cc:357] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "74ce44c58e3944eea01d13eea6b76e61" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40241 } }
I20250903 02:10:32.050588 31319 raft_consensus.cc:383] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:32.050813 31319 raft_consensus.cc:738] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ce44c58e3944eea01d13eea6b76e61, State: Initialized, Role: FOLLOWER
I20250903 02:10:32.051322 31319 consensus_queue.cc:260] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "74ce44c58e3944eea01d13eea6b76e61" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40241 } }
I20250903 02:10:32.051769 31319 raft_consensus.cc:397] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:32.051990 31319 raft_consensus.cc:491] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:32.052217 31319 raft_consensus.cc:3058] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:32.056911 31319 raft_consensus.cc:513] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "74ce44c58e3944eea01d13eea6b76e61" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40241 } }
I20250903 02:10:32.057467 31319 leader_election.cc:304] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74ce44c58e3944eea01d13eea6b76e61; no voters:
I20250903 02:10:32.058624 31319 leader_election.cc:290] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:32.058934 31321 raft_consensus.cc:2802] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:32.060264 31321 raft_consensus.cc:695] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 1 LEADER]: Becoming Leader. State: Replica: 74ce44c58e3944eea01d13eea6b76e61, State: Running, Role: LEADER
I20250903 02:10:32.060441 31309 heartbeater.cc:499] Master 127.27.35.254:45237 was elected leader, sending a full tablet report...
I20250903 02:10:32.061056 31321 consensus_queue.cc:237] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "74ce44c58e3944eea01d13eea6b76e61" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40241 } }
I20250903 02:10:32.061282 31319 ts_tablet_manager.cc:1428] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20250903 02:10:32.065526 31322 tablet_replica.cc:440] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74ce44c58e3944eea01d13eea6b76e61" committed_config { opid_index: -1 peers { permanent_uuid: "74ce44c58e3944eea01d13eea6b76e61" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40241 } } }
I20250903 02:10:32.066053 31322 tablet_replica.cc:443] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61: This TxnStatusTablet replica's current role is: LEADER
I20250903 02:10:32.066083 31323 tablet_replica.cc:440] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61: TxnStatusTablet state changed. Reason: New leader 74ce44c58e3944eea01d13eea6b76e61. Latest consensus state: current_term: 1 leader_uuid: "74ce44c58e3944eea01d13eea6b76e61" committed_config { opid_index: -1 peers { permanent_uuid: "74ce44c58e3944eea01d13eea6b76e61" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40241 } } }
I20250903 02:10:32.066882 31323 tablet_replica.cc:443] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61: This TxnStatusTablet replica's current role is: LEADER
W20250903 02:10:32.068686 31174 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:32.069518 31326 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250903 02:10:32.069986 31326 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250903 02:10:32.072705 31174 catalog_manager.cc:5415] Ignoring report from unknown tablet 11111111111111111111111111111111
I20250903 02:10:32.097570 31274 tablet_service.cc:1468] Processing CreateTablet for tablet 22222222222222222222222222222222 (table=22222222222222222222222222222222_table_name [id=22222222222222222222222222222222_table_id]), partition=RANGE (txn_id, entry_type, identifier) PARTITION (0, -128, "") <= VALUES < (100, -128, "")
I20250903 02:10:32.098603 31274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22222222222222222222222222222222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:32.108907 31319 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61: Bootstrap starting.
I20250903 02:10:32.113579 31319 tablet_bootstrap.cc:654] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:32.118067 31319 tablet_bootstrap.cc:492] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61: No bootstrap required, opened a new log
I20250903 02:10:32.118474 31319 ts_tablet_manager.cc:1397] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20250903 02:10:32.120409 31319 raft_consensus.cc:357] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 peers { permanent_uuid: "74ce44c58e3944eea01d13eea6b76e61" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40241 } }
I20250903 02:10:32.120818 31319 raft_consensus.cc:383] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:32.121043 31319 raft_consensus.cc:738] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ce44c58e3944eea01d13eea6b76e61, State: Initialized, Role: FOLLOWER
I20250903 02:10:32.121559 31319 consensus_queue.cc:260] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "74ce44c58e3944eea01d13eea6b76e61" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40241 } }
I20250903 02:10:32.121987 31319 raft_consensus.cc:397] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:32.122205 31319 raft_consensus.cc:491] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:32.122452 31319 raft_consensus.cc:3058] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:32.127255 31319 raft_consensus.cc:513] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 peers { permanent_uuid: "74ce44c58e3944eea01d13eea6b76e61" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40241 } }
I20250903 02:10:32.127857 31319 leader_election.cc:304] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74ce44c58e3944eea01d13eea6b76e61; no voters:
I20250903 02:10:32.128353 31319 leader_election.cc:290] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:32.128572 31322 raft_consensus.cc:2802] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:32.129078 31322 raft_consensus.cc:695] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 1 LEADER]: Becoming Leader. State: Replica: 74ce44c58e3944eea01d13eea6b76e61, State: Running, Role: LEADER
I20250903 02:10:32.129841 31322 consensus_queue.cc:237] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "74ce44c58e3944eea01d13eea6b76e61" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 40241 } }
I20250903 02:10:32.130168 31319 ts_tablet_manager.cc:1428] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
W20250903 02:10:32.135218 31174 catalog_manager.cc:5415] Ignoring report from unknown tablet 22222222222222222222222222222222
W20250903 02:10:32.139118 31174 catalog_manager.cc:5415] Ignoring report from unknown tablet 22222222222222222222222222222222
I20250903 02:10:32.159473 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:10:32.187574 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:32.188422 27791 tablet_replica.cc:331] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61: stopping tablet replica
I20250903 02:10:32.188925 27791 raft_consensus.cc:2241] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:32.189383 27791 raft_consensus.cc:2270] T 22222222222222222222222222222222 P 74ce44c58e3944eea01d13eea6b76e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:32.192013 27791 tablet_replica.cc:331] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61: stopping tablet replica
I20250903 02:10:32.192512 27791 raft_consensus.cc:2241] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:32.192951 27791 raft_consensus.cc:2270] T 11111111111111111111111111111111 P 74ce44c58e3944eea01d13eea6b76e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:32.205003 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:10:32.215914 27791 master.cc:561] Master@127.27.35.254:45237 shutting down...
I20250903 02:10:32.231600 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:32.232187 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:32.232610 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P a1400cb822e94ff5805e09b4db7e62d3: stopping tablet replica
I20250903 02:10:32.251725 27791 master.cc:583] Master@127.27.35.254:45237 shutdown complete.
[ OK ] TxnStatusTabletManagementTest.TestTabletServerProxyCallErrors (644 ms)
[----------] 4 tests from TxnStatusTabletManagementTest (3402 ms total)
[----------] 2 tests from FailedTabletsAreReplacedITest
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/0
I20250903 02:10:32.276180 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:45597
I20250903 02:10:32.277209 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:32.282351 31335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:32.283476 31336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:32.285339 27791 server_base.cc:1047] running on GCE node
W20250903 02:10:32.285836 31338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:32.286712 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:32.286906 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:32.287066 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865432287048 us; error 0 us; skew 500 ppm
I20250903 02:10:32.287573 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:32.290102 27791 webserver.cc:480] Webserver started at http://127.27.35.254:40431/ using document root <none> and password file <none>
I20250903 02:10:32.290740 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:32.290908 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:32.291169 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:32.292328 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "e3582d1c2f9e4c9596a617c0740064f7"
format_stamp: "Formatted at 2025-09-03 02:10:32 on dist-test-slave-h78g"
I20250903 02:10:32.297034 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250903 02:10:32.300887 31343 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:32.301715 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250903 02:10:32.302014 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "e3582d1c2f9e4c9596a617c0740064f7"
format_stamp: "Formatted at 2025-09-03 02:10:32 on dist-test-slave-h78g"
I20250903 02:10:32.302297 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:32.328290 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:32.329353 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:32.366091 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:45597
I20250903 02:10:32.366187 31394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:45597 every 8 connection(s)
I20250903 02:10:32.369957 31395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:32.380723 31395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7: Bootstrap starting.
I20250903 02:10:32.385061 31395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:32.389081 31395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7: No bootstrap required, opened a new log
I20250903 02:10:32.391074 31395 raft_consensus.cc:357] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3582d1c2f9e4c9596a617c0740064f7" member_type: VOTER }
I20250903 02:10:32.391479 31395 raft_consensus.cc:383] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:32.391701 31395 raft_consensus.cc:738] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3582d1c2f9e4c9596a617c0740064f7, State: Initialized, Role: FOLLOWER
I20250903 02:10:32.392306 31395 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3582d1c2f9e4c9596a617c0740064f7" member_type: VOTER }
I20250903 02:10:32.392743 31395 raft_consensus.cc:397] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:32.392951 31395 raft_consensus.cc:491] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:32.393198 31395 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:32.397912 31395 raft_consensus.cc:513] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3582d1c2f9e4c9596a617c0740064f7" member_type: VOTER }
I20250903 02:10:32.398447 31395 leader_election.cc:304] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3582d1c2f9e4c9596a617c0740064f7; no voters:
I20250903 02:10:32.399533 31395 leader_election.cc:290] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:32.399904 31398 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:32.401290 31398 raft_consensus.cc:695] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 1 LEADER]: Becoming Leader. State: Replica: e3582d1c2f9e4c9596a617c0740064f7, State: Running, Role: LEADER
I20250903 02:10:32.401887 31398 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3582d1c2f9e4c9596a617c0740064f7" member_type: VOTER }
I20250903 02:10:32.402441 31395 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:10:32.404616 31399 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3582d1c2f9e4c9596a617c0740064f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3582d1c2f9e4c9596a617c0740064f7" member_type: VOTER } }
I20250903 02:10:32.404753 31400 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3582d1c2f9e4c9596a617c0740064f7. Latest consensus state: current_term: 1 leader_uuid: "e3582d1c2f9e4c9596a617c0740064f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3582d1c2f9e4c9596a617c0740064f7" member_type: VOTER } }
I20250903 02:10:32.405289 31399 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:32.405452 31400 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:32.408932 31403 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:10:32.416985 31403 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:10:32.417960 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:10:32.426045 31403 catalog_manager.cc:1349] Generated new cluster ID: 13582fde45fe4c78bf2dd56ca69e5f86
I20250903 02:10:32.426337 31403 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:10:32.440210 31403 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:10:32.441496 31403 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:10:32.455969 31403 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7: Generated new TSK 0
I20250903 02:10:32.456583 31403 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:10:32.484566 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:32.490684 31416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:32.492326 31417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:32.495576 31419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:32.496438 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:32.497247 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:32.497473 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:32.497627 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865432497610 us; error 0 us; skew 500 ppm
I20250903 02:10:32.498294 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:32.500809 27791 webserver.cc:480] Webserver started at http://127.27.35.193:42989/ using document root <none> and password file <none>
I20250903 02:10:32.501288 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:32.501463 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:32.501709 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:32.502770 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "680543cc0abf490e9729b2156214c0ec"
format_stamp: "Formatted at 2025-09-03 02:10:32 on dist-test-slave-h78g"
I20250903 02:10:32.507416 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250903 02:10:32.511000 31424 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:32.512003 27791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250903 02:10:32.512336 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "680543cc0abf490e9729b2156214c0ec"
format_stamp: "Formatted at 2025-09-03 02:10:32 on dist-test-slave-h78g"
I20250903 02:10:32.512645 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:32.529871 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:32.530977 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:32.532524 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:10:32.535269 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:32.535491 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:32.535861 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:32.536067 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:32.574760 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:43269
I20250903 02:10:32.574887 31486 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:43269 every 8 connection(s)
I20250903 02:10:32.579315 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:32.587435 31491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:32.588563 31492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:32.593093 31494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:32.593537 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:32.594468 31487 heartbeater.cc:344] Connected to a master server at 127.27.35.254:45597
I20250903 02:10:32.594591 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:32.594864 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:32.594839 31487 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:32.595216 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865432595199 us; error 0 us; skew 500 ppm
I20250903 02:10:32.595870 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:32.595904 31487 heartbeater.cc:507] Master 127.27.35.254:45597 requested a full tablet report, sending...
I20250903 02:10:32.598197 31360 ts_manager.cc:194] Registered new tserver with Master: 680543cc0abf490e9729b2156214c0ec (127.27.35.193:43269)
I20250903 02:10:32.598505 27791 webserver.cc:480] Webserver started at http://127.27.35.194:42593/ using document root <none> and password file <none>
I20250903 02:10:32.598999 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:32.599179 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:32.599421 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:32.600066 31360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35978
I20250903 02:10:32.600523 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-1-root/instance:
uuid: "2953ce2773814920a3932b98962bb038"
format_stamp: "Formatted at 2025-09-03 02:10:32 on dist-test-slave-h78g"
I20250903 02:10:32.605401 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:10:32.608750 31499 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:32.609593 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:10:32.609864 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-1-root
uuid: "2953ce2773814920a3932b98962bb038"
format_stamp: "Formatted at 2025-09-03 02:10:32 on dist-test-slave-h78g"
I20250903 02:10:32.610117 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:32.623729 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:32.624809 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:32.626132 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:10:32.628356 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:32.628546 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:32.628731 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:32.628851 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:32.666695 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.194:34699
I20250903 02:10:32.666795 31561 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.194:34699 every 8 connection(s)
I20250903 02:10:32.671334 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:32.678997 31566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:32.679185 31565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:32.683028 31562 heartbeater.cc:344] Connected to a master server at 127.27.35.254:45597
I20250903 02:10:32.683526 31562 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:32.684468 31562 heartbeater.cc:507] Master 127.27.35.254:45597 requested a full tablet report, sending...
W20250903 02:10:32.684546 31568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:32.684943 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:32.686153 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:32.686409 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:32.686578 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865432686563 us; error 0 us; skew 500 ppm
I20250903 02:10:32.686951 31360 ts_manager.cc:194] Registered new tserver with Master: 2953ce2773814920a3932b98962bb038 (127.27.35.194:34699)
I20250903 02:10:32.687067 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:32.688715 31360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35990
I20250903 02:10:32.689757 27791 webserver.cc:480] Webserver started at http://127.27.35.195:44273/ using document root <none> and password file <none>
I20250903 02:10:32.690271 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:32.690474 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:32.690727 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:32.691865 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-2-root/instance:
uuid: "c306b26448ce438ab8fe1e302d0af012"
format_stamp: "Formatted at 2025-09-03 02:10:32 on dist-test-slave-h78g"
I20250903 02:10:32.696223 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250903 02:10:32.699602 31573 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:32.700533 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 02:10:32.700851 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-2-root
uuid: "c306b26448ce438ab8fe1e302d0af012"
format_stamp: "Formatted at 2025-09-03 02:10:32 on dist-test-slave-h78g"
I20250903 02:10:32.701155 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:32.712436 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:32.713411 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:32.714879 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:10:32.717043 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:32.717244 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:32.717474 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:32.717625 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:32.756057 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.195:43833
I20250903 02:10:32.756150 31635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.195:43833 every 8 connection(s)
I20250903 02:10:32.773615 31636 heartbeater.cc:344] Connected to a master server at 127.27.35.254:45597
I20250903 02:10:32.774009 31636 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:32.774732 31636 heartbeater.cc:507] Master 127.27.35.254:45597 requested a full tablet report, sending...
I20250903 02:10:32.776684 31360 ts_manager.cc:194] Registered new tserver with Master: c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833)
I20250903 02:10:32.777742 27791 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018502893s
I20250903 02:10:32.778020 31360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36004
I20250903 02:10:32.797298 27791 test_util.cc:276] Using random seed: 470433886
I20250903 02:10:32.819345 31360 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36014:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 02:10:32.856686 31601 tablet_service.cc:1468] Processing CreateTablet for tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 (DEFAULT_TABLE table=test-workload [id=d59590a4cbb1426bb2d08f0e38c867cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 02:10:32.858074 31601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4bdeabc4da4fb296514ce4d2ab20e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:32.866425 31527 tablet_service.cc:1468] Processing CreateTablet for tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 (DEFAULT_TABLE table=test-workload [id=d59590a4cbb1426bb2d08f0e38c867cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 02:10:32.867645 31527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4bdeabc4da4fb296514ce4d2ab20e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:32.868062 31452 tablet_service.cc:1468] Processing CreateTablet for tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 (DEFAULT_TABLE table=test-workload [id=d59590a4cbb1426bb2d08f0e38c867cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 02:10:32.869349 31452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4bdeabc4da4fb296514ce4d2ab20e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:32.877245 31656 tablet_bootstrap.cc:492] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Bootstrap starting.
I20250903 02:10:32.885214 31657 tablet_bootstrap.cc:492] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038: Bootstrap starting.
I20250903 02:10:32.885730 31656 tablet_bootstrap.cc:654] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:32.886937 31658 tablet_bootstrap.cc:492] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec: Bootstrap starting.
I20250903 02:10:32.891204 31657 tablet_bootstrap.cc:654] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:32.892786 31656 tablet_bootstrap.cc:492] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: No bootstrap required, opened a new log
I20250903 02:10:32.893283 31656 ts_tablet_manager.cc:1397] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250903 02:10:32.893792 31658 tablet_bootstrap.cc:654] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:32.896163 31656 raft_consensus.cc:357] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:32.897146 31656 raft_consensus.cc:383] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:32.897399 31656 raft_consensus.cc:738] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c306b26448ce438ab8fe1e302d0af012, State: Initialized, Role: FOLLOWER
I20250903 02:10:32.897935 31656 consensus_queue.cc:260] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:32.898357 31657 tablet_bootstrap.cc:492] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038: No bootstrap required, opened a new log
I20250903 02:10:32.898810 31657 ts_tablet_manager.cc:1397] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20250903 02:10:32.900111 31636 heartbeater.cc:499] Master 127.27.35.254:45597 was elected leader, sending a full tablet report...
I20250903 02:10:32.900723 31656 ts_tablet_manager.cc:1428] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250903 02:10:32.901136 31657 raft_consensus.cc:357] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:32.901858 31657 raft_consensus.cc:383] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:32.902190 31657 raft_consensus.cc:738] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2953ce2773814920a3932b98962bb038, State: Initialized, Role: FOLLOWER
I20250903 02:10:32.902534 31658 tablet_bootstrap.cc:492] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec: No bootstrap required, opened a new log
I20250903 02:10:32.902958 31658 ts_tablet_manager.cc:1397] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20250903 02:10:32.902881 31657 consensus_queue.cc:260] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:32.905364 31658 raft_consensus.cc:357] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:32.905867 31658 raft_consensus.cc:383] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:32.906082 31658 raft_consensus.cc:738] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680543cc0abf490e9729b2156214c0ec, State: Initialized, Role: FOLLOWER
I20250903 02:10:32.906661 31658 consensus_queue.cc:260] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:32.909587 31487 heartbeater.cc:499] Master 127.27.35.254:45597 was elected leader, sending a full tablet report...
I20250903 02:10:32.909737 31658 ts_tablet_manager.cc:1428] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250903 02:10:32.911211 31562 heartbeater.cc:499] Master 127.27.35.254:45597 was elected leader, sending a full tablet report...
I20250903 02:10:32.911840 31657 ts_tablet_manager.cc:1428] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20250903 02:10:33.018687 31662 raft_consensus.cc:491] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:10:33.019176 31662 raft_consensus.cc:513] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:33.021240 31662 leader_election.cc:290] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 680543cc0abf490e9729b2156214c0ec (127.27.35.193:43269), 2953ce2773814920a3932b98962bb038 (127.27.35.194:34699)
I20250903 02:10:33.029714 31462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f4bdeabc4da4fb296514ce4d2ab20e5" candidate_uuid: "c306b26448ce438ab8fe1e302d0af012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680543cc0abf490e9729b2156214c0ec" is_pre_election: true
I20250903 02:10:33.030448 31462 raft_consensus.cc:2466] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c306b26448ce438ab8fe1e302d0af012 in term 0.
I20250903 02:10:33.031119 31537 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f4bdeabc4da4fb296514ce4d2ab20e5" candidate_uuid: "c306b26448ce438ab8fe1e302d0af012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2953ce2773814920a3932b98962bb038" is_pre_election: true
I20250903 02:10:33.031407 31576 leader_election.cc:304] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680543cc0abf490e9729b2156214c0ec, c306b26448ce438ab8fe1e302d0af012; no voters:
I20250903 02:10:33.031833 31537 raft_consensus.cc:2466] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c306b26448ce438ab8fe1e302d0af012 in term 0.
I20250903 02:10:33.032264 31662 raft_consensus.cc:2802] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 02:10:33.032598 31662 raft_consensus.cc:491] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:10:33.032882 31662 raft_consensus.cc:3058] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:33.037474 31662 raft_consensus.cc:513] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:33.038825 31662 leader_election.cc:290] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [CANDIDATE]: Term 1 election: Requested vote from peers 680543cc0abf490e9729b2156214c0ec (127.27.35.193:43269), 2953ce2773814920a3932b98962bb038 (127.27.35.194:34699)
I20250903 02:10:33.039610 31462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f4bdeabc4da4fb296514ce4d2ab20e5" candidate_uuid: "c306b26448ce438ab8fe1e302d0af012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680543cc0abf490e9729b2156214c0ec"
I20250903 02:10:33.039798 31537 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f4bdeabc4da4fb296514ce4d2ab20e5" candidate_uuid: "c306b26448ce438ab8fe1e302d0af012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2953ce2773814920a3932b98962bb038"
I20250903 02:10:33.040095 31462 raft_consensus.cc:3058] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:33.040294 31537 raft_consensus.cc:3058] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:33.044620 31462 raft_consensus.cc:2466] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c306b26448ce438ab8fe1e302d0af012 in term 1.
I20250903 02:10:33.044796 31537 raft_consensus.cc:2466] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c306b26448ce438ab8fe1e302d0af012 in term 1.
I20250903 02:10:33.045537 31576 leader_election.cc:304] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680543cc0abf490e9729b2156214c0ec, c306b26448ce438ab8fe1e302d0af012; no voters:
I20250903 02:10:33.046228 31662 raft_consensus.cc:2802] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:33.047037 31662 raft_consensus.cc:695] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 1 LEADER]: Becoming Leader. State: Replica: c306b26448ce438ab8fe1e302d0af012, State: Running, Role: LEADER
I20250903 02:10:33.047676 31662 consensus_queue.cc:237] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:33.054122 31358 catalog_manager.cc:5582] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 reported cstate change: term changed from 0 to 1, leader changed from <none> to c306b26448ce438ab8fe1e302d0af012 (127.27.35.195). New cstate: current_term: 1 leader_uuid: "c306b26448ce438ab8fe1e302d0af012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } }
I20250903 02:10:33.336805 31527 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:33.338333 31452 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:33.338395 31601 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250903 02:10:33.556938 31694 consensus_queue.cc:1035] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e3582d1c2f9e4c9596a617c0740064f7 | 127.27.35.254:45597 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2953ce2773814920a3932b98962bb038 | 127.27.35.194:34699 | HEALTHY | <none> | 0 | 0
680543cc0abf490e9729b2156214c0ec | 127.27.35.193:43269 | HEALTHY | <none> | 0 | 0
c306b26448ce438ab8fe1e302d0af012 | 127.27.35.195:43833 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): RUNNING [LEADER]
680543cc0abf490e9729b2156214c0ec (127.27.35.193:43269): RUNNING
2953ce2773814920a3932b98962bb038 (127.27.35.194:34699): RUNNING
All reported replicas are:
A = c306b26448ce438ab8fe1e302d0af012
B = 680543cc0abf490e9729b2156214c0ec
C = 2953ce2773814920a3932b98962bb038
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250903 02:10:33.574153 31694 consensus_queue.cc:1035] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 02:10:33.575985 27791 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250903 02:10:33.936818 31527 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:33.943702 31601 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250903 02:10:33.945782 31452 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e3582d1c2f9e4c9596a617c0740064f7 | 127.27.35.254:45597 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2953ce2773814920a3932b98962bb038 | 127.27.35.194:34699 | HEALTHY | <none> | 0 | 0
680543cc0abf490e9729b2156214c0ec | 127.27.35.193:43269 | HEALTHY | <none> | 0 | 0
c306b26448ce438ab8fe1e302d0af012 | 127.27.35.195:43833 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250903 02:10:34.138710 27791 log_verifier.cc:126] Checking tablet 3f4bdeabc4da4fb296514ce4d2ab20e5
I20250903 02:10:34.201785 27791 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 3f4bdeabc4da4fb296514ce4d2ab20e5
I20250903 02:10:34.202369 27791 tablet_replica.cc:331] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: stopping tablet replica
I20250903 02:10:34.202940 27791 raft_consensus.cc:2241] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:34.203883 27791 raft_consensus.cc:2270] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:34.414048 31527 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:34.416035 31452 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:34.417462 31601 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e3582d1c2f9e4c9596a617c0740064f7 | 127.27.35.254:45597 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2953ce2773814920a3932b98962bb038 | 127.27.35.194:34699 | HEALTHY | <none> | 0 | 0
680543cc0abf490e9729b2156214c0ec | 127.27.35.193:43269 | HEALTHY | <none> | 0 | 0
c306b26448ce438ab8fe1e302d0af012 | 127.27.35.195:43833 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
680543cc0abf490e9729b2156214c0ec (127.27.35.193:43269): RUNNING
2953ce2773814920a3932b98962bb038 (127.27.35.194:34699): RUNNING
All reported replicas are:
A = c306b26448ce438ab8fe1e302d0af012
B = 680543cc0abf490e9729b2156214c0ec
C = 2953ce2773814920a3932b98962bb038
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250903 02:10:34.626812 27791 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250903 02:10:34.991155 31527 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:34.996865 31601 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:34.997259 31452 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e3582d1c2f9e4c9596a617c0740064f7 | 127.27.35.254:45597 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2953ce2773814920a3932b98962bb038 | 127.27.35.194:34699 | HEALTHY | <none> | 0 | 0
680543cc0abf490e9729b2156214c0ec | 127.27.35.193:43269 | HEALTHY | <none> | 0 | 0
c306b26448ce438ab8fe1e302d0af012 | 127.27.35.195:43833 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): not running [LEADER]
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
680543cc0abf490e9729b2156214c0ec (127.27.35.193:43269): RUNNING
2953ce2773814920a3932b98962bb038 (127.27.35.194:34699): RUNNING
All reported replicas are:
A = c306b26448ce438ab8fe1e302d0af012
B = 680543cc0abf490e9729b2156214c0ec
C = 2953ce2773814920a3932b98962bb038
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A* B C | | | Yes
A | [config not available] | | |
B | A* B C | 1 | -1 | Yes
C | A* B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250903 02:10:35.200706 27791 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250903 02:10:35.510958 31784 raft_consensus.cc:491] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c306b26448ce438ab8fe1e302d0af012)
I20250903 02:10:35.511468 31784 raft_consensus.cc:513] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:35.513521 31784 leader_election.cc:290] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833), 680543cc0abf490e9729b2156214c0ec (127.27.35.193:43269)
I20250903 02:10:35.538175 31789 raft_consensus.cc:491] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c306b26448ce438ab8fe1e302d0af012)
I20250903 02:10:35.538918 31789 raft_consensus.cc:513] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:35.539249 31462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f4bdeabc4da4fb296514ce4d2ab20e5" candidate_uuid: "2953ce2773814920a3932b98962bb038" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "680543cc0abf490e9729b2156214c0ec" is_pre_election: true
I20250903 02:10:35.539788 31611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f4bdeabc4da4fb296514ce4d2ab20e5" candidate_uuid: "2953ce2773814920a3932b98962bb038" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c306b26448ce438ab8fe1e302d0af012" is_pre_election: true
I20250903 02:10:35.540145 31462 raft_consensus.cc:2466] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2953ce2773814920a3932b98962bb038 in term 1.
W20250903 02:10:35.541311 31500 leader_election.cc:343] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): Illegal state: cannot vote while shut down
I20250903 02:10:35.541850 31502 leader_election.cc:304] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2953ce2773814920a3932b98962bb038, 680543cc0abf490e9729b2156214c0ec; no voters: c306b26448ce438ab8fe1e302d0af012
I20250903 02:10:35.542550 31784 raft_consensus.cc:2802] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250903 02:10:35.542928 31784 raft_consensus.cc:491] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c306b26448ce438ab8fe1e302d0af012)
I20250903 02:10:35.543357 31784 raft_consensus.cc:3058] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:10:35.547238 31789 leader_election.cc:290] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833), 2953ce2773814920a3932b98962bb038 (127.27.35.194:34699)
I20250903 02:10:35.550941 31784 raft_consensus.cc:513] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:35.552913 31784 leader_election.cc:290] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [CANDIDATE]: Term 2 election: Requested vote from peers c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833), 680543cc0abf490e9729b2156214c0ec (127.27.35.193:43269)
I20250903 02:10:35.554010 31611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f4bdeabc4da4fb296514ce4d2ab20e5" candidate_uuid: "2953ce2773814920a3932b98962bb038" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c306b26448ce438ab8fe1e302d0af012"
W20250903 02:10:35.555269 31500 leader_election.cc:343] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): Illegal state: cannot vote while shut down
I20250903 02:10:35.560055 31462 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f4bdeabc4da4fb296514ce4d2ab20e5" candidate_uuid: "2953ce2773814920a3932b98962bb038" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "680543cc0abf490e9729b2156214c0ec"
I20250903 02:10:35.560951 31462 raft_consensus.cc:3058] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 1 FOLLOWER]: Advancing to term 2
I20250903 02:10:35.568312 31462 raft_consensus.cc:2466] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2953ce2773814920a3932b98962bb038 in term 2.
I20250903 02:10:35.569584 31502 leader_election.cc:304] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2953ce2773814920a3932b98962bb038, 680543cc0abf490e9729b2156214c0ec; no voters: c306b26448ce438ab8fe1e302d0af012
I20250903 02:10:35.570549 31784 raft_consensus.cc:2802] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 2 FOLLOWER]: Leader election won for term 2
I20250903 02:10:35.571655 31611 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f4bdeabc4da4fb296514ce4d2ab20e5" candidate_uuid: "680543cc0abf490e9729b2156214c0ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c306b26448ce438ab8fe1e302d0af012" is_pre_election: true
W20250903 02:10:35.573202 31425 leader_election.cc:343] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): Illegal state: cannot vote while shut down
I20250903 02:10:35.574280 31784 raft_consensus.cc:695] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 2 LEADER]: Becoming Leader. State: Replica: 2953ce2773814920a3932b98962bb038, State: Running, Role: LEADER
I20250903 02:10:35.574760 31537 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f4bdeabc4da4fb296514ce4d2ab20e5" candidate_uuid: "680543cc0abf490e9729b2156214c0ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2953ce2773814920a3932b98962bb038" is_pre_election: true
I20250903 02:10:35.575269 31784 consensus_queue.cc:237] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:35.576257 31427 leader_election.cc:304] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 680543cc0abf490e9729b2156214c0ec; no voters: 2953ce2773814920a3932b98962bb038, c306b26448ce438ab8fe1e302d0af012
I20250903 02:10:35.577105 31789 raft_consensus.cc:2747] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250903 02:10:35.582978 31360 catalog_manager.cc:5582] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 reported cstate change: term changed from 1 to 2, leader changed from c306b26448ce438ab8fe1e302d0af012 (127.27.35.195) to 2953ce2773814920a3932b98962bb038 (127.27.35.194). New cstate: current_term: 2 leader_uuid: "2953ce2773814920a3932b98962bb038" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } }
I20250903 02:10:35.672721 31527 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250903 02:10:35.674381 31452 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:35.679838 31601 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e3582d1c2f9e4c9596a617c0740064f7 | 127.27.35.254:45597 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2953ce2773814920a3932b98962bb038 | 127.27.35.194:34699 | HEALTHY | <none> | 1 | 0
680543cc0abf490e9729b2156214c0ec | 127.27.35.193:43269 | HEALTHY | <none> | 0 | 0
c306b26448ce438ab8fe1e302d0af012 | 127.27.35.195:43833 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): not running
State: FAILED
Data state: TABLET_DATA_READY
Last status: IO error: INJECTED ERROR: tablet failed
680543cc0abf490e9729b2156214c0ec (127.27.35.193:43269): RUNNING
2953ce2773814920a3932b98962bb038 (127.27.35.194:34699): RUNNING [LEADER]
All reported replicas are:
A = c306b26448ce438ab8fe1e302d0af012
B = 680543cc0abf490e9729b2156214c0ec
C = 2953ce2773814920a3932b98962bb038
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | [config not available] | | |
B | A B C | 2 | -1 | Yes
C | A B C* | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250903 02:10:35.877343 27791 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250903 02:10:35.995862 31462 raft_consensus.cc:1273] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 2 FOLLOWER]: Refusing update from remote peer 2953ce2773814920a3932b98962bb038: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250903 02:10:35.997200 31784 consensus_queue.cc:1035] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250903 02:10:36.006994 31500 consensus_peers.cc:489] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 -> Peer c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): Couldn't send request to peer c306b26448ce438ab8fe1e302d0af012. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20250903 02:10:36.007501 31793 raft_consensus.cc:1010] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038: Attempting to remove follower c306b26448ce438ab8fe1e302d0af012 from the Raft config. Reason: The tablet replica hosted on peer c306b26448ce438ab8fe1e302d0af012 has failed
I20250903 02:10:36.009661 31793 consensus_queue.cc:237] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } }
I20250903 02:10:36.013243 31462 raft_consensus.cc:1273] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 2 FOLLOWER]: Refusing update from remote peer 2953ce2773814920a3932b98962bb038: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250903 02:10:36.014523 31784 consensus_queue.cc:1035] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250903 02:10:36.019788 31793 raft_consensus.cc:2953] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER c306b26448ce438ab8fe1e302d0af012 (127.27.35.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } }
I20250903 02:10:36.021323 31462 raft_consensus.cc:2953] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER c306b26448ce438ab8fe1e302d0af012 (127.27.35.195) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } }
I20250903 02:10:36.029417 31358 catalog_manager.cc:5582] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 reported cstate change: config changed from index -1 to 3, VOTER c306b26448ce438ab8fe1e302d0af012 (127.27.35.195) evicted. New cstate: current_term: 2 leader_uuid: "2953ce2773814920a3932b98962bb038" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } }
I20250903 02:10:36.038034 31601 tablet_service.cc:1515] Processing DeleteTablet for tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 with delete_type TABLET_DATA_TOMBSTONED (TS c306b26448ce438ab8fe1e302d0af012 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:60862
I20250903 02:10:36.039916 31819 ts_tablet_manager.cc:1905] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250903 02:10:36.041234 31537 consensus_queue.cc:237] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } }
W20250903 02:10:36.046765 31500 consensus_peers.cc:489] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 -> Peer c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): Couldn't send request to peer c306b26448ce438ab8fe1e302d0af012. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry.
I20250903 02:10:36.046890 31462 raft_consensus.cc:1273] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 2 FOLLOWER]: Refusing update from remote peer 2953ce2773814920a3932b98962bb038: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250903 02:10:36.048285 31784 consensus_queue.cc:1035] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250903 02:10:36.054031 31793 raft_consensus.cc:2953] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c306b26448ce438ab8fe1e302d0af012 (127.27.35.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } }
I20250903 02:10:36.055178 31819 ts_tablet_manager.cc:1918] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250903 02:10:36.055663 31819 log.cc:1199] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Deleting WAL directory at /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_0.1756865376507568-27791-0/minicluster-data/ts-2-root/wals/3f4bdeabc4da4fb296514ce4d2ab20e5
I20250903 02:10:36.055826 31462 raft_consensus.cc:2953] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c306b26448ce438ab8fe1e302d0af012 (127.27.35.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } }
I20250903 02:10:36.057255 31344 catalog_manager.cc:4928] TS c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 (table test-workload [id=d59590a4cbb1426bb2d08f0e38c867cd]) successfully deleted
I20250903 02:10:36.061326 31346 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20250903 02:10:36.065053 31358 catalog_manager.cc:5582] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 reported cstate change: config changed from index 3 to 4, VOTER c306b26448ce438ab8fe1e302d0af012 (127.27.35.195) added. New cstate: current_term: 2 leader_uuid: "2953ce2773814920a3932b98962bb038" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } } }
I20250903 02:10:36.424963 31527 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250903 02:10:36.433368 31452 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:36.434800 31601 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e3582d1c2f9e4c9596a617c0740064f7 | 127.27.35.254:45597 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2953ce2773814920a3932b98962bb038 | 127.27.35.194:34699 | HEALTHY | <none> | 1 | 0
680543cc0abf490e9729b2156214c0ec | 127.27.35.193:43269 | HEALTHY | <none> | 0 | 0
c306b26448ce438ab8fe1e302d0af012 | 127.27.35.195:43833 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
680543cc0abf490e9729b2156214c0ec (127.27.35.193:43269): RUNNING
2953ce2773814920a3932b98962bb038 (127.27.35.194:34699): RUNNING [LEADER]
c306b26448ce438ab8fe1e302d0af012 (127.27.35.195:43833): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
All reported replicas are:
A = 680543cc0abf490e9729b2156214c0ec
B = 2953ce2773814920a3932b98962bb038
C = c306b26448ce438ab8fe1e302d0af012
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 2 | 4 | Yes
B | A B* C | 2 | 4 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250903 02:10:36.631264 27791 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250903 02:10:36.645401 31849 ts_tablet_manager.cc:927] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Initiating tablet copy from peer 2953ce2773814920a3932b98962bb038 (127.27.35.194:34699)
I20250903 02:10:36.646739 31849 tablet_copy_client.cc:323] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: tablet copy: Beginning tablet copy session from remote peer at address 127.27.35.194:34699
I20250903 02:10:36.648085 31547 tablet_copy_service.cc:140] P 2953ce2773814920a3932b98962bb038: Received BeginTabletCopySession request for tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 from peer c306b26448ce438ab8fe1e302d0af012 ({username='slave'} at 127.0.0.1:60702)
I20250903 02:10:36.648540 31547 tablet_copy_service.cc:161] P 2953ce2773814920a3932b98962bb038: Beginning new tablet copy session on tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 from peer c306b26448ce438ab8fe1e302d0af012 at {username='slave'} at 127.0.0.1:60702: session id = c306b26448ce438ab8fe1e302d0af012-3f4bdeabc4da4fb296514ce4d2ab20e5
I20250903 02:10:36.653483 31547 tablet_copy_source_session.cc:215] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038: Tablet Copy: opened 0 blocks and 1 log segments
I20250903 02:10:36.655568 31849 ts_tablet_manager.cc:1905] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250903 02:10:36.666617 31849 ts_tablet_manager.cc:1918] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20250903 02:10:36.667228 31849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4bdeabc4da4fb296514ce4d2ab20e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:36.672473 31849 tablet_copy_client.cc:806] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: tablet copy: Starting download of 0 data blocks...
I20250903 02:10:36.673030 31849 tablet_copy_client.cc:670] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: tablet copy: Starting download of 1 WAL segments...
I20250903 02:10:36.676167 31849 tablet_copy_client.cc:538] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250903 02:10:36.682585 31849 tablet_bootstrap.cc:492] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Bootstrap starting.
I20250903 02:10:36.699146 31849 tablet_bootstrap.cc:492] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 02:10:36.699959 31849 tablet_bootstrap.cc:492] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Bootstrap complete.
I20250903 02:10:36.700505 31849 ts_tablet_manager.cc:1397] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.004s
I20250903 02:10:36.702365 31849 raft_consensus.cc:357] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } }
I20250903 02:10:36.702888 31849 raft_consensus.cc:738] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c306b26448ce438ab8fe1e302d0af012, State: Initialized, Role: FOLLOWER
I20250903 02:10:36.703475 31849 consensus_queue.cc:260] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "680543cc0abf490e9729b2156214c0ec" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 43269 } } peers { permanent_uuid: "2953ce2773814920a3932b98962bb038" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 34699 } } peers { permanent_uuid: "c306b26448ce438ab8fe1e302d0af012" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 43833 } }
I20250903 02:10:36.705668 31849 ts_tablet_manager.cc:1428] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250903 02:10:36.707257 31547 tablet_copy_service.cc:342] P 2953ce2773814920a3932b98962bb038: Request end of tablet copy session c306b26448ce438ab8fe1e302d0af012-3f4bdeabc4da4fb296514ce4d2ab20e5 received from {username='slave'} at 127.0.0.1:60702
I20250903 02:10:36.707566 31547 tablet_copy_service.cc:434] P 2953ce2773814920a3932b98962bb038: ending tablet copy session c306b26448ce438ab8fe1e302d0af012-3f4bdeabc4da4fb296514ce4d2ab20e5 on tablet 3f4bdeabc4da4fb296514ce4d2ab20e5 with peer c306b26448ce438ab8fe1e302d0af012
I20250903 02:10:37.100746 31611 raft_consensus.cc:1215] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-2.4] Dedup: 2.4->[]
I20250903 02:10:37.378921 31527 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250903 02:10:37.382556 31601 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:37.388545 31452 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
e3582d1c2f9e4c9596a617c0740064f7 | 127.27.35.254:45597 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2953ce2773814920a3932b98962bb038 | 127.27.35.194:34699 | HEALTHY | <none> | 1 | 0
680543cc0abf490e9729b2156214c0ec | 127.27.35.193:43269 | HEALTHY | <none> | 0 | 0
c306b26448ce438ab8fe1e302d0af012 | 127.27.35.195:43833 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-------+-----------------------+-------------------------
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
time_source | system_unsync | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250903 02:10:37.590675 27791 log_verifier.cc:126] Checking tablet 3f4bdeabc4da4fb296514ce4d2ab20e5
I20250903 02:10:37.654373 27791 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 3f4bdeabc4da4fb296514ce4d2ab20e5
I20250903 02:10:37.659432 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:10:37.684163 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:37.684906 27791 tablet_replica.cc:331] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec: stopping tablet replica
I20250903 02:10:37.685586 27791 raft_consensus.cc:2241] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 2 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:37.686267 27791 raft_consensus.cc:2270] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 680543cc0abf490e9729b2156214c0ec [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:37.707062 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:10:37.718343 27791 tablet_server.cc:178] TabletServer@127.27.35.194:0 shutting down...
I20250903 02:10:37.739722 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:37.740495 27791 tablet_replica.cc:331] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038: stopping tablet replica
I20250903 02:10:37.741061 27791 raft_consensus.cc:2241] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 2 LEADER]: Raft consensus shutting down.
I20250903 02:10:37.741987 27791 raft_consensus.cc:2270] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P 2953ce2773814920a3932b98962bb038 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:37.763011 27791 tablet_server.cc:195] TabletServer@127.27.35.194:0 shutdown complete.
I20250903 02:10:37.774750 27791 tablet_server.cc:178] TabletServer@127.27.35.195:0 shutting down...
I20250903 02:10:37.795182 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:37.795914 27791 tablet_replica.cc:331] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012: stopping tablet replica
I20250903 02:10:37.796437 27791 raft_consensus.cc:2241] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:37.796852 27791 raft_consensus.cc:2270] T 3f4bdeabc4da4fb296514ce4d2ab20e5 P c306b26448ce438ab8fe1e302d0af012 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:37.817453 27791 tablet_server.cc:195] TabletServer@127.27.35.195:0 shutdown complete.
I20250903 02:10:37.828095 27791 master.cc:561] Master@127.27.35.254:45597 shutting down...
I20250903 02:10:37.847150 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:37.847831 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:37.848167 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P e3582d1c2f9e4c9596a617c0740064f7: stopping tablet replica
I20250903 02:10:37.867180 27791 master.cc:583] Master@127.27.35.254:45597 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/0 (5621 ms)
[ RUN ] FailedTabletsAreReplacedITest.OneReplica/1
I20250903 02:10:37.897959 27791 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.35.254:41509
I20250903 02:10:37.899049 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:37.904628 31885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:37.905006 31886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:37.907840 31888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:37.907974 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:37.909060 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:37.909245 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:37.909407 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865437909388 us; error 0 us; skew 500 ppm
I20250903 02:10:37.909873 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:37.912554 27791 webserver.cc:480] Webserver started at http://127.27.35.254:33573/ using document root <none> and password file <none>
I20250903 02:10:37.913014 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:37.913183 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:37.913434 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:37.914541 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/master-0-root/instance:
uuid: "3a954a61619a45cc9ca140e8847fdb8e"
format_stamp: "Formatted at 2025-09-03 02:10:37 on dist-test-slave-h78g"
I20250903 02:10:37.918877 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250903 02:10:37.922401 31893 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:37.923295 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 02:10:37.923609 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/master-0-root
uuid: "3a954a61619a45cc9ca140e8847fdb8e"
format_stamp: "Formatted at 2025-09-03 02:10:37 on dist-test-slave-h78g"
I20250903 02:10:37.923943 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:37.941270 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:37.942510 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:37.981211 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.254:41509
I20250903 02:10:37.981391 31944 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.254:41509 every 8 connection(s)
I20250903 02:10:37.985283 31945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:37.996779 31945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e: Bootstrap starting.
I20250903 02:10:38.001639 31945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:38.005980 31945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e: No bootstrap required, opened a new log
I20250903 02:10:38.008230 31945 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a954a61619a45cc9ca140e8847fdb8e" member_type: VOTER }
I20250903 02:10:38.008669 31945 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:38.008899 31945 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a954a61619a45cc9ca140e8847fdb8e, State: Initialized, Role: FOLLOWER
I20250903 02:10:38.009433 31945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a954a61619a45cc9ca140e8847fdb8e" member_type: VOTER }
I20250903 02:10:38.009869 31945 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 02:10:38.010103 31945 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 02:10:38.010346 31945 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:38.015108 31945 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a954a61619a45cc9ca140e8847fdb8e" member_type: VOTER }
I20250903 02:10:38.015910 31945 leader_election.cc:304] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a954a61619a45cc9ca140e8847fdb8e; no voters:
I20250903 02:10:38.017016 31945 leader_election.cc:290] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 02:10:38.017417 31948 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:38.018675 31948 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 1 LEADER]: Becoming Leader. State: Replica: 3a954a61619a45cc9ca140e8847fdb8e, State: Running, Role: LEADER
I20250903 02:10:38.019349 31948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a954a61619a45cc9ca140e8847fdb8e" member_type: VOTER }
I20250903 02:10:38.019960 31945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [sys.catalog]: configured and running, proceeding with master startup.
I20250903 02:10:38.023557 31949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a954a61619a45cc9ca140e8847fdb8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a954a61619a45cc9ca140e8847fdb8e" member_type: VOTER } }
I20250903 02:10:38.024349 31949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:38.025269 31950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a954a61619a45cc9ca140e8847fdb8e. Latest consensus state: current_term: 1 leader_uuid: "3a954a61619a45cc9ca140e8847fdb8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a954a61619a45cc9ca140e8847fdb8e" member_type: VOTER } }
I20250903 02:10:38.025995 31950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [sys.catalog]: This master's current role is: LEADER
I20250903 02:10:38.026261 31954 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 02:10:38.031646 31954 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 02:10:38.035454 27791 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 02:10:38.041519 31954 catalog_manager.cc:1349] Generated new cluster ID: 93527813b2d844faa280ecdb60fa0162
I20250903 02:10:38.041838 31954 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 02:10:38.058632 31954 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 02:10:38.060004 31954 catalog_manager.cc:1506] Loading token signing keys...
I20250903 02:10:38.073625 31954 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e: Generated new TSK 0
I20250903 02:10:38.074292 31954 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 02:10:38.102756 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:38.108618 31966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:38.109548 31967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:38.111016 31969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:38.111430 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:38.112382 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:38.112568 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:38.112739 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865438112721 us; error 0 us; skew 500 ppm
I20250903 02:10:38.113196 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:38.115423 27791 webserver.cc:480] Webserver started at http://127.27.35.193:41825/ using document root <none> and password file <none>
I20250903 02:10:38.115898 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:38.116060 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:38.116266 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:38.117247 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-0-root/instance:
uuid: "6ebd215ce94b470cab7b6e3e36ed9b34"
format_stamp: "Formatted at 2025-09-03 02:10:38 on dist-test-slave-h78g"
I20250903 02:10:38.121443 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:10:38.124519 31974 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.125267 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250903 02:10:38.125543 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-0-root
uuid: "6ebd215ce94b470cab7b6e3e36ed9b34"
format_stamp: "Formatted at 2025-09-03 02:10:38 on dist-test-slave-h78g"
I20250903 02:10:38.125829 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:38.148456 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:38.149549 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:38.150835 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:10:38.153020 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:38.153190 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.153437 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:38.153569 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.193379 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.193:39657
I20250903 02:10:38.193487 32036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.193:39657 every 8 connection(s)
I20250903 02:10:38.198321 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:38.205919 32041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:38.208113 32042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:38.210299 32044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:38.210834 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:38.211789 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:38.211995 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:38.212178 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865438212160 us; error 0 us; skew 500 ppm
I20250903 02:10:38.212697 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:38.214695 32037 heartbeater.cc:344] Connected to a master server at 127.27.35.254:41509
I20250903 02:10:38.215185 32037 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:38.215318 27791 webserver.cc:480] Webserver started at http://127.27.35.194:39723/ using document root <none> and password file <none>
I20250903 02:10:38.215989 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:38.216085 32037 heartbeater.cc:507] Master 127.27.35.254:41509 requested a full tablet report, sending...
I20250903 02:10:38.216245 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:38.216653 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:38.217986 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-1-root/instance:
uuid: "8162a4b2202541a5b70616b9e3a6b51f"
format_stamp: "Formatted at 2025-09-03 02:10:38 on dist-test-slave-h78g"
I20250903 02:10:38.218433 31910 ts_manager.cc:194] Registered new tserver with Master: 6ebd215ce94b470cab7b6e3e36ed9b34 (127.27.35.193:39657)
I20250903 02:10:38.220232 31910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43042
I20250903 02:10:38.222846 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250903 02:10:38.226261 32049 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.226995 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250903 02:10:38.227265 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-1-root
uuid: "8162a4b2202541a5b70616b9e3a6b51f"
format_stamp: "Formatted at 2025-09-03 02:10:38 on dist-test-slave-h78g"
I20250903 02:10:38.227555 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:38.238269 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:38.239467 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:38.240970 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:10:38.243108 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:38.243284 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.243528 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:38.243688 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.281737 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.194:38909
I20250903 02:10:38.281869 32111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.194:38909 every 8 connection(s)
I20250903 02:10:38.286686 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:38.294508 32115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:38.297528 32116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:38.299988 32118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:38.300119 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:38.301033 32112 heartbeater.cc:344] Connected to a master server at 127.27.35.254:41509
I20250903 02:10:38.301472 32112 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:38.301563 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:38.301904 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:38.302109 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865438302093 us; error 0 us; skew 500 ppm
I20250903 02:10:38.302486 32112 heartbeater.cc:507] Master 127.27.35.254:41509 requested a full tablet report, sending...
I20250903 02:10:38.302907 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:38.304965 31910 ts_manager.cc:194] Registered new tserver with Master: 8162a4b2202541a5b70616b9e3a6b51f (127.27.35.194:38909)
I20250903 02:10:38.306000 27791 webserver.cc:480] Webserver started at http://127.27.35.195:39041/ using document root <none> and password file <none>
I20250903 02:10:38.306766 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:38.307008 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:38.306952 31910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43058
I20250903 02:10:38.307498 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:38.308866 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-2-root/instance:
uuid: "b7f29d29d4a74962b68b8c6e04d1b492"
format_stamp: "Formatted at 2025-09-03 02:10:38 on dist-test-slave-h78g"
I20250903 02:10:38.313597 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:10:38.317025 32123 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.317906 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250903 02:10:38.318228 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-2-root
uuid: "b7f29d29d4a74962b68b8c6e04d1b492"
format_stamp: "Formatted at 2025-09-03 02:10:38 on dist-test-slave-h78g"
I20250903 02:10:38.318514 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:38.339653 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:38.340939 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:38.342459 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:10:38.344604 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:38.344817 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.345044 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:38.345196 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.383476 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.195:40815
I20250903 02:10:38.383569 32185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.195:40815 every 8 connection(s)
I20250903 02:10:38.388499 27791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 02:10:38.394702 32189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:38.396961 32190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 02:10:38.398706 32192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 02:10:38.400321 27791 server_base.cc:1047] running on GCE node
I20250903 02:10:38.400805 32186 heartbeater.cc:344] Connected to a master server at 127.27.35.254:41509
I20250903 02:10:38.401309 32186 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:38.401566 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 02:10:38.401837 27791 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 02:10:38.402048 27791 hybrid_clock.cc:648] HybridClock initialized: now 1756865438402027 us; error 0 us; skew 500 ppm
I20250903 02:10:38.402206 32186 heartbeater.cc:507] Master 127.27.35.254:41509 requested a full tablet report, sending...
I20250903 02:10:38.402829 27791 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250903 02:10:38.404810 31910 ts_manager.cc:194] Registered new tserver with Master: b7f29d29d4a74962b68b8c6e04d1b492 (127.27.35.195:40815)
I20250903 02:10:38.405802 27791 webserver.cc:480] Webserver started at http://127.27.35.196:39313/ using document root <none> and password file <none>
I20250903 02:10:38.406335 27791 fs_manager.cc:362] Metadata directory not provided
I20250903 02:10:38.406528 27791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 02:10:38.406483 31910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43062
I20250903 02:10:38.407001 27791 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 02:10:38.408412 27791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-3-root/instance:
uuid: "24c8da0f967c465a9c7ba12953b44299"
format_stamp: "Formatted at 2025-09-03 02:10:38 on dist-test-slave-h78g"
I20250903 02:10:38.413427 27791 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250903 02:10:38.417148 32197 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.418018 27791 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20250903 02:10:38.418329 27791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-3-root
uuid: "24c8da0f967c465a9c7ba12953b44299"
format_stamp: "Formatted at 2025-09-03 02:10:38 on dist-test-slave-h78g"
I20250903 02:10:38.418613 27791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 02:10:38.437661 27791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 02:10:38.438952 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 02:10:38.440582 27791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 02:10:38.442831 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 02:10:38.443037 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.443272 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 02:10:38.443439 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 02:10:38.483973 27791 rpc_server.cc:307] RPC server started. Bound to: 127.27.35.196:37493
I20250903 02:10:38.484056 32259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.35.196:37493 every 8 connection(s)
I20250903 02:10:38.498010 32260 heartbeater.cc:344] Connected to a master server at 127.27.35.254:41509
I20250903 02:10:38.498391 32260 heartbeater.cc:461] Registering TS with master...
I20250903 02:10:38.499068 32260 heartbeater.cc:507] Master 127.27.35.254:41509 requested a full tablet report, sending...
I20250903 02:10:38.500947 31910 ts_manager.cc:194] Registered new tserver with Master: 24c8da0f967c465a9c7ba12953b44299 (127.27.35.196:37493)
I20250903 02:10:38.501395 27791 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.014014159s
I20250903 02:10:38.502388 31910 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60458
I20250903 02:10:38.520960 27791 test_util.cc:276] Using random seed: 476157548
I20250903 02:10:38.542245 31910 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60488:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250903 02:10:38.593497 32151 tablet_service.cc:1468] Processing CreateTablet for tablet 5ba23f7cb65e40e393aca43d2efceb47 (DEFAULT_TABLE table=test-workload [id=7e3c9bafb6eb489bbf64a20d446a880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 02:10:38.593902 32077 tablet_service.cc:1468] Processing CreateTablet for tablet 5ba23f7cb65e40e393aca43d2efceb47 (DEFAULT_TABLE table=test-workload [id=7e3c9bafb6eb489bbf64a20d446a880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 02:10:38.593894 32225 tablet_service.cc:1468] Processing CreateTablet for tablet 5ba23f7cb65e40e393aca43d2efceb47 (DEFAULT_TABLE table=test-workload [id=7e3c9bafb6eb489bbf64a20d446a880e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 02:10:38.594774 32151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba23f7cb65e40e393aca43d2efceb47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:38.595139 32077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba23f7cb65e40e393aca43d2efceb47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:38.595400 32225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba23f7cb65e40e393aca43d2efceb47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:38.609798 32280 tablet_bootstrap.cc:492] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f: Bootstrap starting.
I20250903 02:10:38.613408 32282 tablet_bootstrap.cc:492] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299: Bootstrap starting.
I20250903 02:10:38.614688 32281 tablet_bootstrap.cc:492] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492: Bootstrap starting.
I20250903 02:10:38.618511 32280 tablet_bootstrap.cc:654] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:38.620512 32282 tablet_bootstrap.cc:654] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:38.622686 32281 tablet_bootstrap.cc:654] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492: Neither blocks nor log segments found. Creating new log.
I20250903 02:10:38.625353 32280 tablet_bootstrap.cc:492] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f: No bootstrap required, opened a new log
I20250903 02:10:38.626041 32280 ts_tablet_manager.cc:1397] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20250903 02:10:38.626315 32282 tablet_bootstrap.cc:492] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299: No bootstrap required, opened a new log
I20250903 02:10:38.626776 32282 ts_tablet_manager.cc:1397] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20250903 02:10:38.628155 32281 tablet_bootstrap.cc:492] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492: No bootstrap required, opened a new log
I20250903 02:10:38.628628 32281 ts_tablet_manager.cc:1397] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20250903 02:10:38.629550 32282 raft_consensus.cc:357] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } }
I20250903 02:10:38.630168 32282 raft_consensus.cc:383] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:38.630457 32282 raft_consensus.cc:738] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c8da0f967c465a9c7ba12953b44299, State: Initialized, Role: FOLLOWER
I20250903 02:10:38.630555 32280 raft_consensus.cc:357] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } }
I20250903 02:10:38.631073 32282 consensus_queue.cc:260] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } }
I20250903 02:10:38.631336 32280 raft_consensus.cc:383] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:38.631778 32280 raft_consensus.cc:738] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8162a4b2202541a5b70616b9e3a6b51f, State: Initialized, Role: FOLLOWER
I20250903 02:10:38.631536 32281 raft_consensus.cc:357] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } }
I20250903 02:10:38.632305 32281 raft_consensus.cc:383] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 02:10:38.632699 32281 raft_consensus.cc:738] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f29d29d4a74962b68b8c6e04d1b492, State: Initialized, Role: FOLLOWER
I20250903 02:10:38.632601 32280 consensus_queue.cc:260] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } }
I20250903 02:10:38.633358 32260 heartbeater.cc:499] Master 127.27.35.254:41509 was elected leader, sending a full tablet report...
I20250903 02:10:38.633967 32282 ts_tablet_manager.cc:1428] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250903 02:10:38.633623 32281 consensus_queue.cc:260] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } }
I20250903 02:10:38.636376 32186 heartbeater.cc:499] Master 127.27.35.254:41509 was elected leader, sending a full tablet report...
I20250903 02:10:38.637065 32281 ts_tablet_manager.cc:1428] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20250903 02:10:38.637913 32112 heartbeater.cc:499] Master 127.27.35.254:41509 was elected leader, sending a full tablet report...
I20250903 02:10:38.640770 32280 ts_tablet_manager.cc:1428] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20250903 02:10:38.686991 32287 raft_consensus.cc:491] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 02:10:38.687480 32287 raft_consensus.cc:513] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } }
I20250903 02:10:38.689472 32287 leader_election.cc:290] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24c8da0f967c465a9c7ba12953b44299 (127.27.35.196:37493), b7f29d29d4a74962b68b8c6e04d1b492 (127.27.35.195:40815)
I20250903 02:10:38.698606 32235 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ba23f7cb65e40e393aca43d2efceb47" candidate_uuid: "8162a4b2202541a5b70616b9e3a6b51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c8da0f967c465a9c7ba12953b44299" is_pre_election: true
I20250903 02:10:38.699396 32235 raft_consensus.cc:2466] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8162a4b2202541a5b70616b9e3a6b51f in term 0.
I20250903 02:10:38.700505 32051 leader_election.cc:304] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c8da0f967c465a9c7ba12953b44299, 8162a4b2202541a5b70616b9e3a6b51f; no voters:
I20250903 02:10:38.700582 32161 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ba23f7cb65e40e393aca43d2efceb47" candidate_uuid: "8162a4b2202541a5b70616b9e3a6b51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" is_pre_election: true
I20250903 02:10:38.701222 32287 raft_consensus.cc:2802] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 02:10:38.701220 32161 raft_consensus.cc:2466] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8162a4b2202541a5b70616b9e3a6b51f in term 0.
I20250903 02:10:38.701785 32287 raft_consensus.cc:491] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 02:10:38.702149 32287 raft_consensus.cc:3058] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:38.707643 32287 raft_consensus.cc:513] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } }
I20250903 02:10:38.709048 32287 leader_election.cc:290] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [CANDIDATE]: Term 1 election: Requested vote from peers 24c8da0f967c465a9c7ba12953b44299 (127.27.35.196:37493), b7f29d29d4a74962b68b8c6e04d1b492 (127.27.35.195:40815)
I20250903 02:10:38.709919 32235 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ba23f7cb65e40e393aca43d2efceb47" candidate_uuid: "8162a4b2202541a5b70616b9e3a6b51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c8da0f967c465a9c7ba12953b44299"
I20250903 02:10:38.710023 32161 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ba23f7cb65e40e393aca43d2efceb47" candidate_uuid: "8162a4b2202541a5b70616b9e3a6b51f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f29d29d4a74962b68b8c6e04d1b492"
I20250903 02:10:38.710423 32235 raft_consensus.cc:3058] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:38.710539 32161 raft_consensus.cc:3058] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492 [term 0 FOLLOWER]: Advancing to term 1
I20250903 02:10:38.715416 32235 raft_consensus.cc:2466] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8162a4b2202541a5b70616b9e3a6b51f in term 1.
I20250903 02:10:38.715615 32161 raft_consensus.cc:2466] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8162a4b2202541a5b70616b9e3a6b51f in term 1.
I20250903 02:10:38.716395 32051 leader_election.cc:304] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c8da0f967c465a9c7ba12953b44299, 8162a4b2202541a5b70616b9e3a6b51f; no voters:
I20250903 02:10:38.717057 32287 raft_consensus.cc:2802] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 02:10:38.717906 32287 raft_consensus.cc:695] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 1 LEADER]: Becoming Leader. State: Replica: 8162a4b2202541a5b70616b9e3a6b51f, State: Running, Role: LEADER
I20250903 02:10:38.718535 32287 consensus_queue.cc:237] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } }
I20250903 02:10:38.725364 31908 catalog_manager.cc:5582] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8162a4b2202541a5b70616b9e3a6b51f (127.27.35.194). New cstate: current_term: 1 leader_uuid: "8162a4b2202541a5b70616b9e3a6b51f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } health_report { overall_health: HEALTHY } } }
I20250903 02:10:38.969540 32225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:38.979689 32002 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:38.980414 32077 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250903 02:10:38.985985 32151 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:39.108656 32287 consensus_queue.cc:1035] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f29d29d4a74962b68b8c6e04d1b492" member_type: VOTER last_known_addr { host: "127.27.35.195" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 02:10:39.145653 32287 consensus_queue.cc:1035] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 02:10:39.222952 32037 heartbeater.cc:499] Master 127.27.35.254:41509 was elected leader, sending a full tablet report...
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
3a954a61619a45cc9ca140e8847fdb8e | 127.27.35.254:41509 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
24c8da0f967c465a9c7ba12953b44299 | 127.27.35.196:37493 | HEALTHY | <none> | 0 | 0
6ebd215ce94b470cab7b6e3e36ed9b34 | 127.27.35.193:39657 | HEALTHY | <none> | 0 | 0
8162a4b2202541a5b70616b9e3a6b51f | 127.27.35.194:38909 | HEALTHY | <none> | 1 | 0
b7f29d29d4a74962b68b8c6e04d1b492 | 127.27.35.195:40815 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 5ba23f7cb65e40e393aca43d2efceb47 of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's
24c8da0f967c465a9c7ba12953b44299 (127.27.35.196:37493): RUNNING
b7f29d29d4a74962b68b8c6e04d1b492 (127.27.35.195:40815): RUNNING
8162a4b2202541a5b70616b9e3a6b51f (127.27.35.194:38909): RUNNING [LEADER]
All reported replicas are:
A = 24c8da0f967c465a9c7ba12953b44299
B = b7f29d29d4a74962b68b8c6e04d1b492
C = 8162a4b2202541a5b70616b9e3a6b51f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 1 | -1 | Yes
B | A B C | 1 | -1 | Yes
C | A B C* | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 6ebd215ce94b470cab7b6e3e36ed9b34 | 127.27.35.193:39657 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250903 02:10:39.265870 27791 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250903 02:10:39.624303 32225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:39.631616 32077 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250903 02:10:39.634760 32002 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:39.642995 32151 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
3a954a61619a45cc9ca140e8847fdb8e | 127.27.35.254:41509 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
24c8da0f967c465a9c7ba12953b44299 | 127.27.35.196:37493 | HEALTHY | <none> | 0 | 0
6ebd215ce94b470cab7b6e3e36ed9b34 | 127.27.35.193:39657 | HEALTHY | <none> | 0 | 0
8162a4b2202541a5b70616b9e3a6b51f | 127.27.35.194:38909 | HEALTHY | <none> | 1 | 0
b7f29d29d4a74962b68b8c6e04d1b492 | 127.27.35.195:40815 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 6ebd215ce94b470cab7b6e3e36ed9b34 | 127.27.35.193:39657 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250903 02:10:39.880120 27791 log_verifier.cc:126] Checking tablet 5ba23f7cb65e40e393aca43d2efceb47
I20250903 02:10:39.940539 27791 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5ba23f7cb65e40e393aca43d2efceb47
I20250903 02:10:39.941054 27791 tablet_replica.cc:331] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492: stopping tablet replica
I20250903 02:10:39.941567 27791 raft_consensus.cc:2241] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:39.942009 27791 raft_consensus.cc:2270] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:40.095726 32051 consensus_queue.cc:579] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: The tablet replica hosted on peer b7f29d29d4a74962b68b8c6e04d1b492 has failed
W20250903 02:10:40.096457 32051 consensus_peers.cc:489] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f -> Peer b7f29d29d4a74962b68b8c6e04d1b492 (127.27.35.195:40815): Couldn't send request to peer b7f29d29d4a74962b68b8c6e04d1b492. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: INJECTED ERROR: tablet failed. This is attempt 1: this message will repeat every 5th retry.
I20250903 02:10:40.104256 32087 consensus_queue.cc:237] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } }
I20250903 02:10:40.108829 32235 raft_consensus.cc:1273] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 1 FOLLOWER]: Refusing update from remote peer 8162a4b2202541a5b70616b9e3a6b51f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 02:10:40.110004 32357 consensus_queue.cc:1035] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250903 02:10:40.115379 32290 raft_consensus.cc:2953] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b7f29d29d4a74962b68b8c6e04d1b492 (127.27.35.195) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } }
I20250903 02:10:40.117897 32235 raft_consensus.cc:2953] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER b7f29d29d4a74962b68b8c6e04d1b492 (127.27.35.195) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } }
I20250903 02:10:40.124168 31897 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5ba23f7cb65e40e393aca43d2efceb47 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250903 02:10:40.127194 31907 catalog_manager.cc:5582] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f reported cstate change: config changed from index -1 to 2, VOTER b7f29d29d4a74962b68b8c6e04d1b492 (127.27.35.195) evicted. New cstate: current_term: 1 leader_uuid: "8162a4b2202541a5b70616b9e3a6b51f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } health_report { overall_health: HEALTHY } } }
I20250903 02:10:40.146958 32151 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ba23f7cb65e40e393aca43d2efceb47 with delete_type TABLET_DATA_TOMBSTONED (TS b7f29d29d4a74962b68b8c6e04d1b492 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:56746
I20250903 02:10:40.149026 32382 ts_tablet_manager.cc:1905] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250903 02:10:40.150668 32087 consensus_queue.cc:237] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: NON_VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: true } }
I20250903 02:10:40.169967 32382 ts_tablet_manager.cc:1918] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250903 02:10:40.170707 32382 log.cc:1199] T 5ba23f7cb65e40e393aca43d2efceb47 P b7f29d29d4a74962b68b8c6e04d1b492: Deleting WAL directory at /tmp/dist-test-taskZfrfW4/test-tmp/ts_tablet_manager-itest.0.FailedTabletsAreReplacedITest.OneReplica_1.1756865376507568-27791-0/minicluster-data/ts-2-root/wals/5ba23f7cb65e40e393aca43d2efceb47
I20250903 02:10:40.172720 31895 catalog_manager.cc:4928] TS b7f29d29d4a74962b68b8c6e04d1b492 (127.27.35.195:40815): tablet 5ba23f7cb65e40e393aca43d2efceb47 (table test-workload [id=7e3c9bafb6eb489bbf64a20d446a880e]) successfully deleted
I20250903 02:10:40.174006 32235 raft_consensus.cc:1273] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 1 FOLLOWER]: Refusing update from remote peer 8162a4b2202541a5b70616b9e3a6b51f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250903 02:10:40.175369 32290 consensus_queue.cc:1035] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250903 02:10:40.195098 32357 raft_consensus.cc:2953] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 6ebd215ce94b470cab7b6e3e36ed9b34 (127.27.35.193) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: NON_VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: true } } }
I20250903 02:10:40.203022 32235 raft_consensus.cc:2953] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 6ebd215ce94b470cab7b6e3e36ed9b34 (127.27.35.193) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: NON_VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: true } } }
W20250903 02:10:40.210141 32051 consensus_peers.cc:489] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f -> Peer 6ebd215ce94b470cab7b6e3e36ed9b34 (127.27.35.193:39657): Couldn't send request to peer 6ebd215ce94b470cab7b6e3e36ed9b34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ba23f7cb65e40e393aca43d2efceb47. This is attempt 1: this message will repeat every 5th retry.
I20250903 02:10:40.224067 31907 catalog_manager.cc:5582] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f reported cstate change: config changed from index 2 to 3, NON_VOTER 6ebd215ce94b470cab7b6e3e36ed9b34 (127.27.35.193) added. New cstate: current_term: 1 leader_uuid: "8162a4b2202541a5b70616b9e3a6b51f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: NON_VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250903 02:10:40.236498 31897 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5ba23f7cb65e40e393aca43d2efceb47 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250903 02:10:40.260993 32077 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250903 02:10:40.267234 32225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:40.273380 32002 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:40.289022 32151 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
3a954a61619a45cc9ca140e8847fdb8e | 127.27.35.254:41509 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
24c8da0f967c465a9c7ba12953b44299 | 127.27.35.196:37493 | HEALTHY | <none> | 0 | 0
6ebd215ce94b470cab7b6e3e36ed9b34 | 127.27.35.193:39657 | HEALTHY | <none> | 0 | 0
8162a4b2202541a5b70616b9e3a6b51f | 127.27.35.194:38909 | HEALTHY | <none> | 1 | 0
b7f29d29d4a74962b68b8c6e04d1b492 | 127.27.35.195:40815 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 5ba23f7cb65e40e393aca43d2efceb47 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
24c8da0f967c465a9c7ba12953b44299 (127.27.35.196:37493): RUNNING
b7f29d29d4a74962b68b8c6e04d1b492 (127.27.35.195:40815): not running
State: FAILED
Data state: TABLET_DATA_TOMBSTONED
Last status: Deleted tablet blocks from disk
8162a4b2202541a5b70616b9e3a6b51f (127.27.35.194:38909): RUNNING [LEADER]
All reported replicas are:
A = 24c8da0f967c465a9c7ba12953b44299
B = b7f29d29d4a74962b68b8c6e04d1b492
C = 8162a4b2202541a5b70616b9e3a6b51f
D = 6ebd215ce94b470cab7b6e3e36ed9b34
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A C* D~ | 1 | 3 | Yes
B | [config not available] | | |
C | A C* D~ | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | 6ebd215ce94b470cab7b6e3e36ed9b34 | 127.27.35.193:39657 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250903 02:10:40.500912 27791 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250903 02:10:40.721776 32402 ts_tablet_manager.cc:927] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: Initiating tablet copy from peer 8162a4b2202541a5b70616b9e3a6b51f (127.27.35.194:38909)
I20250903 02:10:40.723467 32402 tablet_copy_client.cc:323] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: tablet copy: Beginning tablet copy session from remote peer at address 127.27.35.194:38909
I20250903 02:10:40.733770 32097 tablet_copy_service.cc:140] P 8162a4b2202541a5b70616b9e3a6b51f: Received BeginTabletCopySession request for tablet 5ba23f7cb65e40e393aca43d2efceb47 from peer 6ebd215ce94b470cab7b6e3e36ed9b34 ({username='slave'} at 127.0.0.1:50966)
I20250903 02:10:40.734416 32097 tablet_copy_service.cc:161] P 8162a4b2202541a5b70616b9e3a6b51f: Beginning new tablet copy session on tablet 5ba23f7cb65e40e393aca43d2efceb47 from peer 6ebd215ce94b470cab7b6e3e36ed9b34 at {username='slave'} at 127.0.0.1:50966: session id = 6ebd215ce94b470cab7b6e3e36ed9b34-5ba23f7cb65e40e393aca43d2efceb47
I20250903 02:10:40.740926 32097 tablet_copy_source_session.cc:215] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f: Tablet Copy: opened 0 blocks and 1 log segments
I20250903 02:10:40.743479 32402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ba23f7cb65e40e393aca43d2efceb47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 02:10:40.756202 32402 tablet_copy_client.cc:806] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: tablet copy: Starting download of 0 data blocks...
I20250903 02:10:40.756767 32402 tablet_copy_client.cc:670] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: tablet copy: Starting download of 1 WAL segments...
I20250903 02:10:40.760210 32402 tablet_copy_client.cc:538] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250903 02:10:40.766700 32402 tablet_bootstrap.cc:492] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: Bootstrap starting.
I20250903 02:10:40.782940 32402 tablet_bootstrap.cc:492] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 02:10:40.783681 32402 tablet_bootstrap.cc:492] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: Bootstrap complete.
I20250903 02:10:40.784281 32402 ts_tablet_manager.cc:1397] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.007s
I20250903 02:10:40.786258 32402 raft_consensus.cc:357] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: NON_VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: true } }
I20250903 02:10:40.786826 32402 raft_consensus.cc:738] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ebd215ce94b470cab7b6e3e36ed9b34, State: Initialized, Role: LEARNER
I20250903 02:10:40.787362 32402 consensus_queue.cc:260] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: NON_VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: true } }
I20250903 02:10:40.790038 32402 ts_tablet_manager.cc:1428] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250903 02:10:40.795095 32097 tablet_copy_service.cc:342] P 8162a4b2202541a5b70616b9e3a6b51f: Request end of tablet copy session 6ebd215ce94b470cab7b6e3e36ed9b34-5ba23f7cb65e40e393aca43d2efceb47 received from {username='slave'} at 127.0.0.1:50966
I20250903 02:10:40.795562 32097 tablet_copy_service.cc:434] P 8162a4b2202541a5b70616b9e3a6b51f: ending tablet copy session 6ebd215ce94b470cab7b6e3e36ed9b34-5ba23f7cb65e40e393aca43d2efceb47 on tablet 5ba23f7cb65e40e393aca43d2efceb47 with peer 6ebd215ce94b470cab7b6e3e36ed9b34
I20250903 02:10:40.885928 32225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:40.888823 32077 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250903 02:10:40.892789 32151 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:40.898442 32002 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
3a954a61619a45cc9ca140e8847fdb8e | 127.27.35.254:41509 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
24c8da0f967c465a9c7ba12953b44299 | 127.27.35.196:37493 | HEALTHY | <none> | 0 | 0
6ebd215ce94b470cab7b6e3e36ed9b34 | 127.27.35.193:39657 | HEALTHY | <none> | 0 | 0
8162a4b2202541a5b70616b9e3a6b51f | 127.27.35.194:38909 | HEALTHY | <none> | 1 | 0
b7f29d29d4a74962b68b8c6e04d1b492 | 127.27.35.195:40815 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 5ba23f7cb65e40e393aca43d2efceb47 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
24c8da0f967c465a9c7ba12953b44299 (127.27.35.196:37493): RUNNING
8162a4b2202541a5b70616b9e3a6b51f (127.27.35.194:38909): RUNNING [LEADER]
6ebd215ce94b470cab7b6e3e36ed9b34 (127.27.35.193:39657): RUNNING [NONVOTER]
All reported replicas are:
A = 24c8da0f967c465a9c7ba12953b44299
B = 8162a4b2202541a5b70616b9e3a6b51f
C = 6ebd215ce94b470cab7b6e3e36ed9b34
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C~ | | | Yes
A | A B* C~ | 1 | 3 | Yes
B | A B* C~ | 1 | 3 | Yes
C | A B C~ | 1 | 3 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | b7f29d29d4a74962b68b8c6e04d1b492 | 127.27.35.195:40815 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250903 02:10:41.123220 27791 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250903 02:10:41.219170 32012 raft_consensus.cc:1215] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20250903 02:10:41.573364 32225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:41.577484 32002 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:41.580420 32077 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250903 02:10:41.583886 32151 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250903 02:10:41.706960 32430 raft_consensus.cc:1062] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f: attempting to promote NON_VOTER 6ebd215ce94b470cab7b6e3e36ed9b34 to VOTER
I20250903 02:10:41.709213 32430 consensus_queue.cc:237] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: false } }
I20250903 02:10:41.716925 32012 raft_consensus.cc:1273] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34 [term 1 LEARNER]: Refusing update from remote peer 8162a4b2202541a5b70616b9e3a6b51f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250903 02:10:41.718565 32290 consensus_queue.cc:1035] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250903 02:10:41.726209 32235 raft_consensus.cc:1273] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 1 FOLLOWER]: Refusing update from remote peer 8162a4b2202541a5b70616b9e3a6b51f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250903 02:10:41.727862 32290 consensus_queue.cc:1035] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250903 02:10:41.747257 32235 raft_consensus.cc:2953] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 6ebd215ce94b470cab7b6e3e36ed9b34 (127.27.35.193) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: false } } }
I20250903 02:10:41.746035 32290 raft_consensus.cc:2953] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 6ebd215ce94b470cab7b6e3e36ed9b34 (127.27.35.193) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: false } } }
I20250903 02:10:41.762033 31908 catalog_manager.cc:5582] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f reported cstate change: config changed from index 3 to 4, 6ebd215ce94b470cab7b6e3e36ed9b34 (127.27.35.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8162a4b2202541a5b70616b9e3a6b51f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250903 02:10:41.774821 32012 raft_consensus.cc:2953] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 6ebd215ce94b470cab7b6e3e36ed9b34 (127.27.35.193) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "24c8da0f967c465a9c7ba12953b44299" member_type: VOTER last_known_addr { host: "127.27.35.196" port: 37493 } } peers { permanent_uuid: "8162a4b2202541a5b70616b9e3a6b51f" member_type: VOTER last_known_addr { host: "127.27.35.194" port: 38909 } } peers { permanent_uuid: "6ebd215ce94b470cab7b6e3e36ed9b34" member_type: VOTER last_known_addr { host: "127.27.35.193" port: 39657 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
3a954a61619a45cc9ca140e8847fdb8e | 127.27.35.254:41509 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 1 server(s) checked
time_source | system_unsync | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
24c8da0f967c465a9c7ba12953b44299 | 127.27.35.196:37493 | HEALTHY | <none> | 0 | 0
6ebd215ce94b470cab7b6e3e36ed9b34 | 127.27.35.193:39657 | HEALTHY | <none> | 0 | 0
8162a4b2202541a5b70616b9e3a6b51f | 127.27.35.194:38909 | HEALTHY | <none> | 1 | 0
b7f29d29d4a74962b68b8c6e04d1b492 | 127.27.35.195:40815 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------+--------+-------------------------
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 4 server(s) checked
time_source | system_unsync | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+---------------------+---------------
Small | b7f29d29d4a74962b68b8c6e04d1b492 | 127.27.35.195:40815 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250903 02:10:41.863373 27791 log_verifier.cc:126] Checking tablet 5ba23f7cb65e40e393aca43d2efceb47
I20250903 02:10:41.923075 27791 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 5ba23f7cb65e40e393aca43d2efceb47
I20250903 02:10:41.925061 27791 tablet_server.cc:178] TabletServer@127.27.35.193:0 shutting down...
I20250903 02:10:41.948475 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:41.949120 27791 tablet_replica.cc:331] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34: stopping tablet replica
I20250903 02:10:41.949631 27791 raft_consensus.cc:2241] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:41.950060 27791 raft_consensus.cc:2270] T 5ba23f7cb65e40e393aca43d2efceb47 P 6ebd215ce94b470cab7b6e3e36ed9b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:41.969969 27791 tablet_server.cc:195] TabletServer@127.27.35.193:0 shutdown complete.
I20250903 02:10:41.979274 27791 tablet_server.cc:178] TabletServer@127.27.35.194:0 shutting down...
I20250903 02:10:41.998320 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:41.998909 27791 tablet_replica.cc:331] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f: stopping tablet replica
I20250903 02:10:41.999397 27791 raft_consensus.cc:2241] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:42.000152 27791 raft_consensus.cc:2270] T 5ba23f7cb65e40e393aca43d2efceb47 P 8162a4b2202541a5b70616b9e3a6b51f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:42.010243 27791 tablet_server.cc:195] TabletServer@127.27.35.194:0 shutdown complete.
I20250903 02:10:42.019618 27791 tablet_server.cc:178] TabletServer@127.27.35.195:0 shutting down...
I20250903 02:10:42.040848 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:42.057983 27791 tablet_server.cc:195] TabletServer@127.27.35.195:0 shutdown complete.
I20250903 02:10:42.067227 27791 tablet_server.cc:178] TabletServer@127.27.35.196:0 shutting down...
I20250903 02:10:42.087355 27791 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 02:10:42.088228 27791 tablet_replica.cc:331] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299: stopping tablet replica
I20250903 02:10:42.088925 27791 raft_consensus.cc:2241] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250903 02:10:42.089656 27791 raft_consensus.cc:2270] T 5ba23f7cb65e40e393aca43d2efceb47 P 24c8da0f967c465a9c7ba12953b44299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:42.109817 27791 tablet_server.cc:195] TabletServer@127.27.35.196:0 shutdown complete.
I20250903 02:10:42.119104 27791 master.cc:561] Master@127.27.35.254:41509 shutting down...
I20250903 02:10:42.136415 27791 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 1 LEADER]: Raft consensus shutting down.
I20250903 02:10:42.136960 27791 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 02:10:42.137269 27791 tablet_replica.cc:331] T 00000000000000000000000000000000 P 3a954a61619a45cc9ca140e8847fdb8e: stopping tablet replica
I20250903 02:10:42.154902 27791 master.cc:583] Master@127.27.35.254:41509 shutdown complete.
[ OK ] FailedTabletsAreReplacedITest.OneReplica/1 (4284 ms)
[----------] 2 tests from FailedTabletsAreReplacedITest (9906 ms total)
[----------] Global test environment tear-down
[==========] 13 tests from 4 test suites ran. (65508 ms total)
[ PASSED ] 11 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] LeadershipChangeReportingTest.TestReportStatsDuringLeadershipChange
[ FAILED ] 1 test, listed below:
[ FAILED ] TsTabletManagerITest.TestTableStats
1 FAILED TEST
I20250903 02:10:42.189424 27791 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:788: suppressed but not reported on 3 messages since previous log ~10 seconds ago
I20250903 02:10:42.189749 27791 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 76 messages since previous log ~57 seconds ago
I20250903 02:10:42.189942 27791 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 39 messages since previous log ~19 seconds ago
I20250903 02:10:42.190254 27791 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 9 messages since previous log ~14 seconds ago